Nov 03 02:58:11.271358 ip-10-0-136-7 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 02:58:11.271370 ip-10-0-136-7 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 02:58:11.271377 ip-10-0-136-7 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 02:58:11.271631 ip-10-0-136-7 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 02:58:21.490383 ip-10-0-136-7 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 02:58:21.490400 ip-10-0-136-7 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3b1f17848d0142249d35ed20e8f9ee23 -- Nov 03 03:00:30.703155 ip-10-0-136-7 systemd[1]: Starting Kubernetes Kubelet... Nov 03 03:00:31.122094 ip-10-0-136-7 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:31.122094 ip-10-0-136-7 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 03:00:31.122094 ip-10-0-136-7 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:31.122094 ip-10-0-136-7 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:31.127897 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127786 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.127897 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127896 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.127897 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127899 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.127897 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127902 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127906 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127909 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127912 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127915 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127919 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127922 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127925 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127928 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127930 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127933 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127936 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127938 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127942 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127944 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127947 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127949 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127953 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127956 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127959 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.128025 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127962 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127964 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127967 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127970 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127972 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127975 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127979 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127981 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127984 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127987 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127989 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127992 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127995 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.127997 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128001 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128004 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128006 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128010 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128013 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128016 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.128497 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128019 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128021 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128024 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128027 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128030 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128033 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128036 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128039 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128042 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128045 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128049 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128051 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128054 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128056 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128059 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128062 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128064 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128067 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128069 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.129029 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128072 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128076 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128080 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128084 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128087 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128091 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128094 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128097 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128101 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128104 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128106 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128109 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128112 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128114 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128117 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128120 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128122 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128125 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128127 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.129472 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128130 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128132 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128135 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128137 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128140 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128143 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128146 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128149 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128152 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128154 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128156 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128159 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128161 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128164 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128166 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128172 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128177 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128180 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128183 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.129952 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128186 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128189 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128192 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128195 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128197 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.128200 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129043 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129050 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129053 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129056 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129059 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129062 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129064 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129067 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129070 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129073 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129076 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129078 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129081 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129090 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.130412 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129092 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129095 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129098 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129100 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129104 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129108 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129111 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129114 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129117 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129119 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129122 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129126 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129128 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129132 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129135 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129137 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129140 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129142 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129145 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.130907 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129148 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129150 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129153 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129156 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129158 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129161 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129163 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129166 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129168 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129171 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129173 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129175 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129178 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129186 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129189 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129191 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129194 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129197 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129199 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129201 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.131531 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129204 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129206 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129209 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129211 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129214 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129217 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129220 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129223 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129226 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129228 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129231 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129234 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129236 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129239 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129243 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129245 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129248 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129250 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129253 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129255 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.132037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129257 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129260 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129262 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129265 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129267 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129270 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129278 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129281 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129283 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129286 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129288 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129291 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129293 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129296 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129298 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129301 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129306 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129310 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129314 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.132519 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129317 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129320 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129323 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129326 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129328 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129331 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129334 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129336 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129339 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129341 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129344 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129346 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129349 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129351 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129937 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129943 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129947 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129949 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129952 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.132994 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129961 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129964 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129967 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129969 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129972 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129975 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129977 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129981 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129984 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129987 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129989 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129993 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129995 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.129998 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130001 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130004 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130006 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130009 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130012 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130015 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.133448 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130018 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130020 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130022 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130025 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130028 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130030 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130033 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130035 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130038 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130040 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130042 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130045 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130047 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130051 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130053 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130056 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130058 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130061 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130063 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130066 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.133950 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130068 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130071 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130074 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130077 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130080 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130083 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130085 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130088 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130092 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130095 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130098 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130101 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130104 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130106 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130109 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130111 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130113 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130116 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130118 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.134427 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130121 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130124 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130126 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130128 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130131 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130134 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130136 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130139 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130143 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130147 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130151 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130154 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130157 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130159 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130162 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130164 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130168 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130170 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130173 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.134891 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130175 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130177 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130180 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130183 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130185 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130188 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130190 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130193 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130195 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130198 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130200 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130203 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130205 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130207 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130210 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130212 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130215 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130217 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130220 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.135338 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130222 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130225 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130228 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.130250 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131151 2571 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131161 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131168 2571 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131172 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131183 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131187 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131192 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131197 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131201 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131204 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131207 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131210 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131213 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131216 2571 flags.go:64] FLAG: --cgroup-root="" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131219 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131222 2571 flags.go:64] FLAG: --client-ca-file="" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131225 2571 flags.go:64] FLAG: --cloud-provider="external" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131228 2571 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131234 2571 flags.go:64] FLAG: --cluster-domain="" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131238 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 03:00:31.135776 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131241 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131244 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131247 2571 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131251 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131254 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131257 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131260 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131263 2571 flags.go:64] FLAG: --contention-profiling="false" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131266 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131269 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131272 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131276 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131281 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131284 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131287 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131290 2571 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131294 2571 flags.go:64] FLAG: --enable-server="true" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131297 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131302 2571 flags.go:64] FLAG: --event-burst="100" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131305 2571 flags.go:64] FLAG: --event-qps="50" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131308 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131311 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131314 2571 flags.go:64] FLAG: --eviction-hard="" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131318 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 03:00:31.136340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131321 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131324 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131327 2571 flags.go:64] FLAG: --eviction-soft="" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131330 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131333 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131336 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131339 2571 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131342 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131345 2571 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131347 2571 flags.go:64] FLAG: --feature-gates="" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131352 2571 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131355 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131358 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131362 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131365 2571 flags.go:64] FLAG: --healthz-port="10248" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131369 2571 flags.go:64] FLAG: --help="false" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131372 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131375 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131378 2571 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131381 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131385 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131388 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131391 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 03:00:31.136981 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131394 2571 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131397 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131399 2571 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131403 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131406 2571 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131409 2571 flags.go:64] FLAG: --kube-reserved="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131412 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131414 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131417 2571 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131420 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131423 2571 flags.go:64] FLAG: --lock-file="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131426 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131428 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131431 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131437 2571 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131440 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131443 2571 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131446 2571 flags.go:64] FLAG: --logging-format="text" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131449 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131452 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131455 2571 flags.go:64] FLAG: --manifest-url="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131458 2571 flags.go:64] FLAG: --manifest-url-header="" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131463 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131466 2571 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131470 2571 flags.go:64] FLAG: --max-pods="110" Nov 03 03:00:31.137555 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131473 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131476 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131479 2571 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131482 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131485 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131488 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131491 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131499 2571 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131502 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131505 2571 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131512 2571 flags.go:64] FLAG: --pod-cidr="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131515 2571 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131518 2571 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131521 2571 flags.go:64] FLAG: --pods-per-core="0" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131524 2571 flags.go:64] FLAG: --port="10250" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131527 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131530 2571 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d535352506dc27d7" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131533 2571 flags.go:64] FLAG: --qos-reserved="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131536 2571 flags.go:64] FLAG: --read-only-port="10255" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131539 2571 flags.go:64] FLAG: --register-node="true" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131543 2571 flags.go:64] FLAG: --register-with-taints="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131549 2571 flags.go:64] FLAG: --registry-burst="10" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131552 2571 flags.go:64] FLAG: --registry-qps="5" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131555 2571 flags.go:64] FLAG: --reserved-cpus="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131558 2571 flags.go:64] FLAG: --reserved-memory="" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131561 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 03:00:31.138163 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131564 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131567 2571 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131570 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131573 2571 flags.go:64] FLAG: --runonce="false" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131576 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131579 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131582 2571 flags.go:64] FLAG: --seccomp-default="false" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131585 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131588 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131591 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131594 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131598 2571 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131607 2571 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131610 2571 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131613 2571 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131616 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131621 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131624 2571 flags.go:64] FLAG: --system-cgroups="" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131627 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131632 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131635 2571 flags.go:64] FLAG: --tls-cert-file="" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131638 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131645 2571 flags.go:64] FLAG: --tls-min-version="" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131648 2571 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131651 2571 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 03:00:31.138778 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131653 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131656 2571 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131663 2571 flags.go:64] FLAG: --v="2" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131668 2571 flags.go:64] FLAG: --version="false" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131672 2571 flags.go:64] FLAG: --vmodule="" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131676 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.131680 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131860 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131864 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131868 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131871 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131874 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131877 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131880 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131882 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131885 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131888 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131891 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131893 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131896 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131900 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.139424 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131903 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131906 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131910 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131913 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131916 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131918 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131921 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131924 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131926 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131929 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131931 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131934 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131937 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131941 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131944 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131946 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131949 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131952 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131955 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131957 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.139947 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131960 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131963 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131965 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131968 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131970 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131973 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131975 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131978 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131980 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131983 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131986 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131988 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131991 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131994 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.131998 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132001 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132003 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132006 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132008 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.140459 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132011 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132013 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132016 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132019 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132022 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132024 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132028 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132031 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132033 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132036 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132039 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132042 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132045 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132048 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132050 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132054 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132059 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132062 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132065 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.140919 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132068 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132072 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132076 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132079 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132082 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132084 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132087 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132091 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132095 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132098 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132100 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132103 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132106 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132108 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132111 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132114 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132116 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132119 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132122 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.141383 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132126 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132129 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132131 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132134 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132137 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132140 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132142 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132145 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132148 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132150 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132153 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132155 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132158 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132161 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132163 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.132193 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:31.142135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132229 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132233 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132236 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132238 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132241 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132246 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132248 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132251 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132254 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132257 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132260 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132262 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132265 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132267 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132271 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132273 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132276 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132280 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132283 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132285 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.142753 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132288 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132290 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132293 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132295 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132298 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132300 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132303 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132305 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132307 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132310 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132312 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132315 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132318 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132320 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132323 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132326 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132328 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132332 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132335 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.143306 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132337 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132340 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132342 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132345 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132347 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132349 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132352 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132354 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132357 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132359 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132363 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132365 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132368 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132370 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132374 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132378 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132382 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132386 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132388 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.143748 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132392 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132396 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132399 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132402 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132405 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132407 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132410 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132412 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132415 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132417 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132420 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132423 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132426 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132429 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132432 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132434 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132438 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132440 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132443 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132445 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.144243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132448 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132450 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132453 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132457 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132459 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132462 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132465 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132467 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132470 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132472 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132475 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132477 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132480 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132482 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132485 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132487 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132490 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132492 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132495 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132497 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.144712 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132500 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132502 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132505 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132509 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132512 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132514 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132519 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.132522 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.140463 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.140479 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.140496 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.140501 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140564 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140569 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140572 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140575 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140579 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140582 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140584 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.145199 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140589 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140594 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140598 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140601 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140604 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140607 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140610 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140612 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140615 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140618 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140620 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140623 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140626 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140628 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140630 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140633 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140636 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140638 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140641 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140644 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.145663 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140647 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140649 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140653 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140655 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140658 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140661 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140663 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140666 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140669 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140671 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140674 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140676 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140679 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140681 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140684 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140686 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140689 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140691 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140694 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.146153 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140696 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140699 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140701 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140704 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140706 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140708 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140711 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140714 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140716 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140719 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140722 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140724 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140727 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140730 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140733 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140736 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140739 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140741 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140744 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140746 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.146633 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140749 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140751 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140754 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140756 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140759 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140762 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140764 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140767 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140769 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140772 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140774 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140777 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140779 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140783 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140787 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140790 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140793 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140795 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140798 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.147243 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140800 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140803 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140806 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140808 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140811 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140814 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140817 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140819 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140822 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140824 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140827 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140830 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140832 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140835 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140837 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140840 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140861 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140864 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140867 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140870 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.147715 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.140872 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.140906 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141052 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141058 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141061 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141063 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141066 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141068 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141071 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141073 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141076 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141079 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141081 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141083 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141086 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141089 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.148214 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141092 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141095 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141097 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141100 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141103 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141106 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141108 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141111 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141114 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141117 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141121 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141124 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141127 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141129 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141132 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141134 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141137 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141140 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141143 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141145 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.148636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141148 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141150 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141153 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141155 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141159 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141163 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141166 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141168 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141171 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141173 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141176 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141179 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141181 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141184 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141187 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141189 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141192 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141195 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141197 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.149135 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141200 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141203 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141205 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141208 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141210 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141213 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141215 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141217 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141220 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141222 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141225 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141227 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141230 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141233 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141235 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141238 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141240 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141243 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141246 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141248 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.149575 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141251 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141253 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141255 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141258 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141261 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141263 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141266 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141269 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141272 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141274 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141277 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141279 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141282 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141284 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.141336 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142170 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142181 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142186 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142192 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.150149 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142197 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142202 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142206 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142211 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142216 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142221 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142225 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142229 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142234 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142243 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142248 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142252 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142257 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142262 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.142336 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142458 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:31.150649 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142467 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142472 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142477 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142481 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142486 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142491 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142495 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142499 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142504 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142508 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142513 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142522 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142538 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142542 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142546 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142550 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142556 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142560 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142564 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142569 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:31.151094 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142574 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142579 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142583 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142588 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142596 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142600 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142603 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142606 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142609 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142612 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142615 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142618 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142620 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142623 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142627 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142630 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142635 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142638 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142641 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:31.151580 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142644 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142648 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142650 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142653 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142655 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142658 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142661 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142665 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142668 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142670 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142676 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142678 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142681 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142687 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142689 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142692 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142695 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142698 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142700 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:31.152072 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142703 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142706 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142709 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142712 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142717 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142720 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142724 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142727 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142730 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142732 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142735 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142738 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142740 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142743 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142745 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142748 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142751 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142756 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142759 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142762 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:31.152554 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142767 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142770 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142773 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142776 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142779 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142782 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142786 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142790 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142797 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142800 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142803 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.142805 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143032 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143045 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143051 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143058 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143063 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143068 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143073 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:31.153164 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143078 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143082 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143088 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143094 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143099 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143103 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143108 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:31.143113 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.143227 2571 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.145454 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.146504 2571 server.go:1014] "Starting client certificate rotation" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.146601 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.146638 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.148566 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:00:31.153676 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.150067 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:00:31.162630 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.162613 2571 log.go:25] "Validated CRI v1 runtime API" Nov 03 03:00:31.164987 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.164973 2571 log.go:25] "Validated CRI v1 image API" Nov 03 03:00:31.165801 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.165778 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 03:00:31.169886 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.169861 2571 fs.go:135] Filesystem UUIDs: map[46cba2f0-ec2b-403e-8dd9-5313e7d4b767:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9160006a-86fd-4a30-8214-c03e07d39e64:/dev/nvme0n1p4] Nov 03 03:00:31.169950 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.169886 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 03:00:31.175105 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.174989 2571 manager.go:217] Machine: {Timestamp:2026-11-03 03:00:31.173600526 +0000 UTC m=+0.352642895 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121082 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea70e0b26cef491289f802562cde9 SystemUUID:ec2ea70e-0b26-cef4-9128-9f802562cde9 BootID:3b1f1784-8d01-4224-9d35-ed20e8f9ee23 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:a8:91:f7:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:a8:91:f7:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:39:66:95:1a:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 03:00:31.175105 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.175105 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 03:00:31.175212 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.175197 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 03:00:31.178204 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178176 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 03:00:31.178340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178205 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-7.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 03:00:31.178382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178343 2571 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 03:00:31.178382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178351 2571 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 03:00:31.178430 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178387 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:00:31.178430 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.178407 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:00:31.179255 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.179244 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 03:00:31.183169 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.183155 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 03:00:31.183304 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.183294 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 03:00:31.184000 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.183989 2571 kubelet.go:493] "Attempting to sync node with API server" Nov 03 03:00:31.184041 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.184003 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 03:00:31.184041 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.184014 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 03:00:31.184041 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.184039 2571 kubelet.go:405] "Adding apiserver pod source" Nov 03 03:00:31.184165 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.184050 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 03:00:31.184415 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.184403 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:00:31.186709 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.186692 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 03:00:31.193018 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.192998 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:00:31.193827 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.193813 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 03:00:31.193893 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.193863 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 03:00:31.194026 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194014 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 03:00:31.194066 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194030 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 03:00:31.194066 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194036 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 03:00:31.194066 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194042 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 03:00:31.194066 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194048 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 03:00:31.194066 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194054 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 03:00:31.194735 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194726 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 03:00:31.194764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194736 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 03:00:31.194764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194743 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 03:00:31.194764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194749 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 03:00:31.194764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194758 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 03:00:31.194764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.194766 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 03:00:31.195253 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.195243 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 03:00:31.195287 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.195254 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 03:00:31.199002 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.198989 2571 server.go:1263] "Started kubelet" Nov 03 03:00:31.199130 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.199066 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 03:00:31.199197 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.199115 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 03:00:31.199197 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.199192 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 03:00:31.199796 ip-10-0-136-7 systemd[1]: Started Kubernetes Kubelet. Nov 03 03:00:31.200393 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.200152 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 03:00:31.201697 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.201682 2571 server.go:317] "Adding debug handlers to kubelet server" Nov 03 03:00:31.205064 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.205047 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 03:00:31.205290 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.205275 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 03:00:31.205951 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.205934 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 03:00:31.206135 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.206110 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:31.206593 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.206578 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 03:00:31.206593 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.206578 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 03:00:31.206763 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.206622 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 03:00:31.206763 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.206711 2571 reconstruct.go:97] "Volume reconstruction finished" Nov 03 03:00:31.206763 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.206721 2571 reconciler.go:29] "Reconciler: start to sync state" Nov 03 03:00:31.207045 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207029 2571 factory.go:55] Registering systemd factory Nov 03 03:00:31.207083 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207050 2571 factory.go:223] Registration of the systemd container factory successfully Nov 03 03:00:31.207299 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207284 2571 factory.go:153] Registering CRI-O factory Nov 03 03:00:31.207340 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207304 2571 factory.go:223] Registration of the crio container factory successfully Nov 03 03:00:31.207387 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207363 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 03:00:31.207387 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207380 2571 factory.go:103] Registering Raw factory Nov 03 03:00:31.207444 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.207393 2571 manager.go:1199] Started watching for new ooms in manager Nov 03 03:00:31.208326 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.208311 2571 manager.go:319] Starting recovery of all containers Nov 03 03:00:31.215336 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.215293 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 03:00:31.220802 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.220780 2571 manager.go:324] Recovery completed Nov 03 03:00:31.223252 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.223214 2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 03 03:00:31.226418 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.226401 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.228242 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228217 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.228320 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228255 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.228320 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228266 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.228884 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228866 2571 cpu_manager.go:226] "Starting" policy="none" Nov 03 03:00:31.228932 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228885 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.228962 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228883 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 03:00:31.228989 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.228978 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 03:00:31.229522 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.229503 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.229613 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.229539 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.229613 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.229553 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.231107 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.231096 2571 policy_none.go:50] "Start" Nov 03 03:00:31.231143 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.231113 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 03:00:31.231143 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.231123 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 03:00:31.233138 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.233128 2571 policy_none.go:44] "Start" Nov 03 03:00:31.276718 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.276528 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 03:00:31.280791 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.280772 2571 manager.go:341] "Starting Device Plugin manager" Nov 03 03:00:31.280932 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.280803 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 03:00:31.280932 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.280812 2571 server.go:87] "Starting device plugin registration server" Nov 03 03:00:31.281083 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.281070 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 03:00:31.281132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.281083 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 03:00:31.281211 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.281199 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 03:00:31.281323 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.281306 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 03:00:31.281323 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.281318 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 03:00:31.281875 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.281840 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 03:00:31.281957 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.281903 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:31.343986 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.343959 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 03:00:31.343986 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.343988 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 03:00:31.346156 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.344009 2571 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 03:00:31.346156 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.344053 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 03:00:31.381606 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.381521 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.382502 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.382486 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.382587 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.382517 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.382587 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.382528 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.382587 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.382556 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.403740 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.403710 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-7.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 03:00:31.403994 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.403761 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-7.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 03:00:31.404562 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.403802 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-7.us-west-1.compute.internal.18e469c1742c3a79 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-7.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-7.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:31.198960249 +0000 UTC m=+0.378002611,LastTimestamp:2026-11-03 03:00:31.198960249 +0000 UTC m=+0.378002611,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-7.us-west-1.compute.internal,}" Nov 03 03:00:31.444323 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.444281 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal"] Nov 03 03:00:31.444406 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.444362 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.445301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.445281 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.445372 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.445316 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.445372 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.445326 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.447455 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.447443 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.447638 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.447621 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.447680 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.447657 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.448238 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448219 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.448347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448253 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.448347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448262 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.448347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448288 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.448347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448299 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.448347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.448268 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.450364 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.450350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.450435 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.450376 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.451283 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.451268 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.451344 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.451300 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.451344 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.451317 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.461725 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.461707 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 03:00:31.461813 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.461777 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6r2hv" Nov 03 03:00:31.475397 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.475376 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.477508 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.477490 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.508634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.508601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.508634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.508632 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.508828 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.508649 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.512749 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.512724 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 03:00:31.512749 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.512724 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-7.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 03:00:31.512888 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.512743 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 03:00:31.512926 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.512810 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-7.us-west-1.compute.internal.18e469c175eb0588 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-7.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-7.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-7.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:31.228241288 +0000 UTC m=+0.407283650,LastTimestamp:2026-11-03 03:00:31.228241288 +0000 UTC m=+0.407283650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-7.us-west-1.compute.internal,}" Nov 03 03:00:31.569164 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.569131 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:31.570293 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.570278 2571 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.609862 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.609939 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609884 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.609939 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609913 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.610001 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609939 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.610001 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.610061 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.609994 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.708551 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.708469 2571 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.771042 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.771011 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:31.772067 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.772048 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:31.772149 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.772083 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:31.772149 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.772094 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:31.772149 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.772126 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.779212 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.779185 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.781335 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.781320 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.833266 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:31.833240 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:31.833377 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.833269 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-7.us-west-1.compute.internal\": node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:31.953609 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:31.953577 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.054389 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.054299 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.146087 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.146052 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 03:00:32.146633 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.146250 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 03:00:32.146633 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.146256 2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a058e290254ff44d893eba1331bf9de6-33cafa47f6ec0d31.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-7.us-west-1.compute.internal.18e469c175eb77ff\": read tcp 10.0.136.7:51660->18.116.9.183:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-7.us-west-1.compute.internal.18e469c175eb77ff default 5030 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-7.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-7.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-7.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:31 +0000 UTC,LastTimestamp:2026-11-03 03:00:31.382532453 +0000 UTC m=+0.561574815,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-7.us-west-1.compute.internal,}" Nov 03 03:00:32.155365 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.155343 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.256160 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.256125 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.356837 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.356766 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.456863 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.456821 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.466892 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.466841 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:31 +0000 UTC" deadline="2028-07-22 23:30:07.34758839 +0000 UTC" Nov 03 03:00:32.466892 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.466889 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15068h29m34.880704901s" Nov 03 03:00:32.470007 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.469991 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:32.470934 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.470915 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:32.471004 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.470948 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:32.471004 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.470958 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:32.557916 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.557883 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.658985 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.658893 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.759508 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.759471 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.776034 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.775999 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:32.860304 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.860274 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:32.874909 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:32.874874 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e862e4bcfcee2fc388ea5939653511.slice/crio-fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa WatchSource:0}: Error finding container fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa: Status 404 returned error can't find the container with id fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa Nov 03 03:00:32.880299 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:32.880281 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:00:32.920156 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:32.920125 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50507705f9f63f8e4597fb6868abd9a.slice/crio-1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a WatchSource:0}: Error finding container 1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a: Status 404 returned error can't find the container with id 1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a Nov 03 03:00:32.960755 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:32.960707 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:33.035671 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.035645 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:33.061633 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:33.061605 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:33.162609 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:33.162567 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:00:33.187600 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.187543 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:33.207056 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.207017 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 03:00:33.207139 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.207032 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:33.273289 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.273256 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:00:33.274352 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.274336 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:00:33.293282 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.293256 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:33.337114 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.337090 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:00:33.349686 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.349625 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerStarted","Data":"1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a"} Nov 03 03:00:33.350836 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.350814 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"b7e862e4bcfcee2fc388ea5939653511","Type":"ContainerStarted","Data":"fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa"} Nov 03 03:00:33.379084 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.379054 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 03:00:33.379227 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:33.379119 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-75skm" Nov 03 03:00:34.185646 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.185597 2571 apiserver.go:52] "Watching apiserver" Nov 03 03:00:34.352773 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.352742 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 03:00:34.353200 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.353174 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-tdbsm","openshift-multus/multus-additional-cni-plugins-qwcqm","openshift-multus/multus-tgpmj","openshift-network-operator/iptables-alerter-4gfm4","kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-4c62k","openshift-ovn-kubernetes/ovnkube-node-gvtt8","openshift-network-diagnostics/network-check-target-jk4nq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb","openshift-cluster-node-tuning-operator/tuned-vg788","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-4vzrm","kube-system/konnectivity-agent-v499x","openshift-dns/node-resolver-ncxp7"] Nov 03 03:00:34.356262 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.356238 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.358585 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.358557 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:34.358696 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.358641 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:34.360946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.360924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.363315 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.363292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.365704 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.365688 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.368160 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.368143 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.368233 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.368201 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:34.371042 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.371022 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.373412 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.373390 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.375990 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.375969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.378332 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.378312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.380908 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.380887 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.382125 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.382102 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-04-23 13:46:49.140884918 +0000 UTC" Nov 03 03:00:34.382215 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.382126 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12898h46m14.758763192s" Nov 03 03:00:34.383382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.383368 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.383451 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.383428 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:34.408859 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.408836 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 03:00:34.415170 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.415150 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.415521 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.415503 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.415863 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.415847 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 03:00:34.415929 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.415851 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 03:00:34.415929 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.415872 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\"" Nov 03 03:00:34.431176 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431145 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.431315 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.431315 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431219 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.431315 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431289 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.431481 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.431481 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.431481 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431384 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.431481 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431439 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.431481 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.431664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.431664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.431664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.431664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431638 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.431832 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431665 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.431832 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431711 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xxh\" (UniqueName: \"kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.431832 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.431832 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.431832 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431800 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431955 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.431980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.432100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432077 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftswn\" (UniqueName: \"kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432113 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432142 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432207 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432290 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432315 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432369 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432424 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.432495 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432480 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432591 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9m9\" (UniqueName: \"kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432836 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjttr\" (UniqueName: \"kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxs5\" (UniqueName: \"kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.432981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433065 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433111 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffgl\" (UniqueName: \"kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433139 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433169 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433195 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433315 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258g4\" (UniqueName: \"kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433365 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.433664 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433389 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z929b\" (UniqueName: \"kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433435 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433458 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433474 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433494 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95g7q\" (UniqueName: \"kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433570 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.434241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.433610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.454049 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.453976 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454042 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454079 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454088 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454096 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454042 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454126 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454142 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454159 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454062 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454178 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\"" Nov 03 03:00:34.454191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.454197 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\"" Nov 03 03:00:34.468364 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.468334 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 03:00:34.468644 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.468628 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.468816 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.468797 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.469541 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469525 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\"" Nov 03 03:00:34.469541 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469535 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\"" Nov 03 03:00:34.469666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469554 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\"" Nov 03 03:00:34.469666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469618 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 03:00:34.469666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469626 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 03:00:34.469666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469617 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.469666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469652 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.469889 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469708 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 03:00:34.469972 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.469945 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 03:00:34.470056 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.470025 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.470056 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.470035 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\"" Nov 03 03:00:34.470056 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.470041 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:34.470168 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.470023 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:34.470168 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.470069 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 03:00:34.507860 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.507817 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\"" Nov 03 03:00:34.508333 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.508319 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\"" Nov 03 03:00:34.534117 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534225 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534225 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534225 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534204 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffgl\" (UniqueName: \"kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534380 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534390 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258g4\" (UniqueName: \"kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534559 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z929b\" (UniqueName: \"kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534621 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534675 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534686 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.534711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95g7q\" (UniqueName: \"kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534780 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534862 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534730 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534952 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534732 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.534989 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.535448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535023 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535048 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535051 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535079 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535150 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535200 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535214 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535262 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535308 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.536280 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.535412 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xxh\" (UniqueName: \"kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.535479 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:35.035456562 +0000 UTC m=+4.214498934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535680 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535749 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535810 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535841 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535855 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535860 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535858 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.537136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.535995 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.535997 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536067 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.536079 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:00:35.036061718 +0000 UTC m=+4.215104071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536078 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536133 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536178 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536204 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftswn\" (UniqueName: \"kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536226 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536357 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.537858 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536402 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536439 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536473 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536493 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536527 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536566 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536580 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536614 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536643 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536661 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536667 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.538575 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536698 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536699 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536698 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536715 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536765 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536815 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536822 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9m9\" (UniqueName: \"kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536853 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.536967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.539301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537097 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537103 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537114 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537128 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537222 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjttr\" (UniqueName: \"kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537335 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxs5\" (UniqueName: \"kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537365 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537447 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.540132 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537456 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.540653 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.540653 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.540653 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:34.540653 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.537679 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.541257 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.541234 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.541342 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.541289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.541423 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.541398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.541619 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.541605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.599598 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.599561 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:34.599598 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.599591 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:34.599598 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.599607 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:34.599932 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:34.599686 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:35.09966492 +0000 UTC m=+4.278707286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:34.602170 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.602143 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z929b\" (UniqueName: \"kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.603040 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.602650 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258g4\" (UniqueName: \"kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.603040 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.602664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95g7q\" (UniqueName: \"kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:34.603040 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.602771 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffgl\" (UniqueName: \"kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.603040 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.602992 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftswn\" (UniqueName: \"kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.607906 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.607807 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9m9\" (UniqueName: \"kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.608133 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.608111 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxs5\" (UniqueName: \"kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.608624 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.608607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjttr\" (UniqueName: \"kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.608982 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.608966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xxh\" (UniqueName: \"kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.685667 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.685622 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tgpmj" Nov 03 03:00:34.686788 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.686768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:00:34.692519 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.692493 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:00:34.693784 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.693760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:00:34.698742 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.698716 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e95865_15f3_4a5d_9e0d_47a5c172ddde.slice/crio-53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a WatchSource:0}: Error finding container 53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a: Status 404 returned error can't find the container with id 53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a Nov 03 03:00:34.700448 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.700429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:00:34.704442 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.704422 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:34.704809 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.704788 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6141f3_f144_46ab_86c0_d780fcb0129e.slice/crio-edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799 WatchSource:0}: Error finding container edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799: Status 404 returned error can't find the container with id edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799 Nov 03 03:00:34.707084 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.707042 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75ca389_0c41_4d4f_9788_da6f8bb2cc70.slice/crio-da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e WatchSource:0}: Error finding container da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e: Status 404 returned error can't find the container with id da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e Nov 03 03:00:34.710698 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.710672 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5ce847_feef_4cdf_b403_18eaa916c05c.slice/crio-26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508 WatchSource:0}: Error finding container 26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508: Status 404 returned error can't find the container with id 26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508 Nov 03 03:00:34.715068 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.715039 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1e4336_ab3b_4e70_a11c_8566fbc03510.slice/crio-fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f WatchSource:0}: Error finding container fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f: Status 404 returned error can't find the container with id fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f Nov 03 03:00:34.716037 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.716017 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b95e773_2411_4fbc_b0f1_0e5fbde12efb.slice/crio-98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36 WatchSource:0}: Error finding container 98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36: Status 404 returned error can't find the container with id 98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36 Nov 03 03:00:34.723301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.723281 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:00:34.725694 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.725676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:00:34.728492 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:34.728474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:00:34.733586 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.733564 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162d9031_a839_4af7_8ba4_973b900c618e.slice/crio-414e4dd6ffeee16cfeb675d3feb02eb598166346d82db142f8a0418aadf5950b WatchSource:0}: Error finding container 414e4dd6ffeee16cfeb675d3feb02eb598166346d82db142f8a0418aadf5950b: Status 404 returned error can't find the container with id 414e4dd6ffeee16cfeb675d3feb02eb598166346d82db142f8a0418aadf5950b Nov 03 03:00:34.736835 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.736791 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce981b0_12d5_4738_8dfe_2683c56f1a07.slice/crio-bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb WatchSource:0}: Error finding container bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb: Status 404 returned error can't find the container with id bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb Nov 03 03:00:34.740273 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:00:34.740237 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4794a03d_e0ff_401e_b267_70361f49b0a0.slice/crio-b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558 WatchSource:0}: Error finding container b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558: Status 404 returned error can't find the container with id b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558 Nov 03 03:00:35.040301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.040212 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:35.040301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.040249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:35.040536 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.040358 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:35.040536 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.040381 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:35.040536 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.040414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.040400839 +0000 UTC m=+5.219443203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:35.040536 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.040459 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.040441219 +0000 UTC m=+5.219483591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:35.141614 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.141575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:35.141783 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.141714 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:35.141783 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.141731 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:35.141783 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.141740 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:35.141904 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:35.141803 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.141785067 +0000 UTC m=+5.320827422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:35.355182 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.355091 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tgpmj" event={"ID":"d6e95865-15f3-4a5d-9e0d-47a5c172ddde","Type":"ContainerStarted","Data":"53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a"} Nov 03 03:00:35.356347 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.356319 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ncxp7" event={"ID":"fce981b0-12d5-4738-8dfe-2683c56f1a07","Type":"ContainerStarted","Data":"bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb"} Nov 03 03:00:35.358068 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.358041 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v499x" event={"ID":"4b95e773-2411-4fbc-b0f1-0e5fbde12efb","Type":"ContainerStarted","Data":"98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36"} Nov 03 03:00:35.359467 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.359439 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" event={"ID":"ee1e4336-ab3b-4e70-a11c-8566fbc03510","Type":"ContainerStarted","Data":"fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f"} Nov 03 03:00:35.360643 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.360621 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdbsm" event={"ID":"bf5ce847-feef-4cdf-b403-18eaa916c05c","Type":"ContainerStarted","Data":"26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508"} Nov 03 03:00:35.361985 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.361965 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerStarted","Data":"b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558"} Nov 03 03:00:35.363140 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.363111 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vg788" event={"ID":"162d9031-a839-4af7-8ba4-973b900c618e","Type":"ContainerStarted","Data":"414e4dd6ffeee16cfeb675d3feb02eb598166346d82db142f8a0418aadf5950b"} Nov 03 03:00:35.364328 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.364307 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e"} Nov 03 03:00:35.365367 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.365348 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfm4" event={"ID":"de6141f3-f144-46ab-86c0-d780fcb0129e","Type":"ContainerStarted","Data":"edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799"} Nov 03 03:00:35.383126 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.383075 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-05-03 12:23:57.048256171 +0000 UTC" Nov 03 03:00:35.383126 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:35.383119 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13137h23m21.665139143s" Nov 03 03:00:36.053953 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.053905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:36.054173 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.053969 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:36.054173 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.054104 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:36.054173 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.054167 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:00:38.05414994 +0000 UTC m=+7.233192294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:36.054354 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.054234 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:36.054354 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.054269 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:38.054257008 +0000 UTC m=+7.233299365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:36.155047 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.155006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:36.155258 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.155239 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:36.155319 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.155270 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:36.155319 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.155285 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:36.155439 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.155347 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:38.155327511 +0000 UTC m=+7.334369874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:36.345125 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.345036 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:36.345323 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.345205 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:36.345323 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.345311 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:36.345445 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.345402 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:36.345497 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:36.345464 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:36.345561 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:36.345543 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:38.070216 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.070143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:38.070216 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.070197 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:38.070770 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.070335 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:38.070770 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.070400 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:00:42.070380453 +0000 UTC m=+11.249422825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:38.070932 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.070823 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:38.070932 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.070881 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:42.070863643 +0000 UTC m=+11.249906019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:38.173241 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.172822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:38.173241 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.173082 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:38.173241 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.173112 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:38.173241 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.173128 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:38.173241 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.173197 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:42.173178224 +0000 UTC m=+11.352220601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.345172 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.345212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.345314 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:38.345336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.345423 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:38.345811 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:38.345493 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:39.417026 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:39.416492 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="361c6ad97554fbf33f8476e06b9c2549c0dcecf22a1926c5b417bed3b4b10686" exitCode=0 Nov 03 03:00:39.417026 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:39.416648 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"361c6ad97554fbf33f8476e06b9c2549c0dcecf22a1926c5b417bed3b4b10686"} Nov 03 03:00:40.344923 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.344828 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:40.345080 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:40.344980 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:40.345366 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.345344 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:40.345472 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:40.345439 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:40.345537 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.345496 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:40.345590 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:40.345564 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:40.423245 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.423069 2571 generic.go:356] "Generic (PLEG): container finished" podID="d50507705f9f63f8e4597fb6868abd9a" containerID="11bcdfef753f14737486560a132ccd4c58a96e4b62574fab608e824938d61060" exitCode=0 Nov 03 03:00:40.423245 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.423163 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerDied","Data":"11bcdfef753f14737486560a132ccd4c58a96e4b62574fab608e824938d61060"} Nov 03 03:00:40.427882 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:40.427778 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v499x" event={"ID":"4b95e773-2411-4fbc-b0f1-0e5fbde12efb","Type":"ContainerStarted","Data":"1a857803a71a00ca94c9be844969e2698b52f82630ea5562b11875a2b47f6ffe"} Nov 03 03:00:41.435748 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:41.435711 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"b7e862e4bcfcee2fc388ea5939653511","Type":"ContainerStarted","Data":"7cb35b4a9db4e6b2b2db6a2ab570cf2b8de78b5780beba3dafc1dd957a588240"} Nov 03 03:00:41.553991 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:41.553925 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v499x" podStartSLOduration=4.633375378 podStartE2EDuration="9.5539032s" totalImagesPullingTime="4.920527822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:40.551831792 +0000 UTC m=+9.730874337" watchObservedRunningTime="2026-11-03 03:00:41.5539032 +0000 UTC m=+10.732945577" Nov 03 03:00:41.669579 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:41.669509 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" podStartSLOduration=8.66949088 podStartE2EDuration="8.66949088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:41.55338435 +0000 UTC m=+10.732426721" watchObservedRunningTime="2026-11-03 03:00:41.66949088 +0000 UTC m=+10.848533252" Nov 03 03:00:42.138775 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.138581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:42.138775 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.138630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:42.138775 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.138768 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:42.139102 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.138826 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.138808975 +0000 UTC m=+19.317851338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:42.139102 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.138909 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:42.139102 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.138943 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.138932258 +0000 UTC m=+19.317974619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:42.240002 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.239960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:42.240191 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.240174 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:42.240252 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.240196 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:42.240252 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.240208 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:42.240355 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.240267 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.240249041 +0000 UTC m=+19.419291393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:42.344666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.344589 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:42.344826 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.344735 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:42.344826 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.344809 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:42.344948 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.344895 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:42.344948 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.344942 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:42.345039 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:42.345006 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:42.440751 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.439925 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerStarted","Data":"ee57a2b65703cb1be46b8147d6fc24cfb9da74f8c890ef65108d213a58b5d04d"} Nov 03 03:00:42.444026 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.443990 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" event={"ID":"ee1e4336-ab3b-4e70-a11c-8566fbc03510","Type":"ContainerStarted","Data":"cbd96b6196985a999f5388df956a45d6c5ff9bbed80525edd452c5cd6ba3062e"} Nov 03 03:00:42.446732 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.446706 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdbsm" event={"ID":"bf5ce847-feef-4cdf-b403-18eaa916c05c","Type":"ContainerStarted","Data":"26189afe8f160824aebcd44757c6a0771b1d3d001fcd1ca05e268bde8e1da5c5"} Nov 03 03:00:42.672695 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.671827 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" podStartSLOduration=9.67181054 podStartE2EDuration="9.67181054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:42.557128164 +0000 UTC m=+11.736170537" watchObservedRunningTime="2026-11-03 03:00:42.67181054 +0000 UTC m=+11.850852911" Nov 03 03:00:42.672695 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:42.672211 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tdbsm" podStartSLOduration=3.489149836 podStartE2EDuration="10.672202583s" totalImagesPullingTime="7.183052747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:42.671373964 +0000 UTC m=+11.850416336" watchObservedRunningTime="2026-11-03 03:00:42.672202583 +0000 UTC m=+11.851244958" Nov 03 03:00:43.153059 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:43.152497 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:43.153625 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:43.153462 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:43.452313 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:43.452027 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tgpmj" event={"ID":"d6e95865-15f3-4a5d-9e0d-47a5c172ddde","Type":"ContainerStarted","Data":"68a0c42f6e0cc69efd34492f2aefe9fbc2b63f13a47d530f57394ae34321a7e9"} Nov 03 03:00:44.345048 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.345017 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:44.345186 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.345060 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:44.345186 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:44.345154 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:44.345320 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.345228 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:44.345320 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:44.345245 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:44.345447 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:44.345324 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:44.450566 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.450391 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 03:00:44.455376 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.455348 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ncxp7" event={"ID":"fce981b0-12d5-4738-8dfe-2683c56f1a07","Type":"ContainerStarted","Data":"32bca93d54475ea43ddce99dd8d37fac6daae20875f9b5af9f8c8ecf9cb5367f"} Nov 03 03:00:44.457753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.457725 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" event={"ID":"ee1e4336-ab3b-4e70-a11c-8566fbc03510","Type":"ContainerStarted","Data":"62e120fe0a405742f35faee99a2f4b4fac26eb010b08aa65a5d6e421d76de253"} Nov 03 03:00:44.459684 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.459664 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="d687f62c7ea7674a7439203ae6c8f375f4cf32e425e10b999046761a1a375fe4" exitCode=0 Nov 03 03:00:44.459774 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.459756 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"d687f62c7ea7674a7439203ae6c8f375f4cf32e425e10b999046761a1a375fe4"} Nov 03 03:00:44.583740 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.582980 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ncxp7" podStartSLOduration=3.571900231 podStartE2EDuration="12.582965383s" totalImagesPullingTime="9.011065152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:44.582906635 +0000 UTC m=+13.761949008" watchObservedRunningTime="2026-11-03 03:00:44.582965383 +0000 UTC m=+13.762007755" Nov 03 03:00:44.583740 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:44.583197 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tgpmj" podStartSLOduration=4.140054529 podStartE2EDuration="12.583192274s" totalImagesPullingTime="8.443137745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:43.574903925 +0000 UTC m=+12.753946296" watchObservedRunningTime="2026-11-03 03:00:44.583192274 +0000 UTC m=+13.762234644" Nov 03 03:00:45.288747 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.288637 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:00:44.45056304Z","UUID":"107e607b-a55d-4a14-82ee-4809ec263de5","Handler":null,"Name":"","Endpoint":""} Nov 03 03:00:45.290070 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.290046 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 03:00:45.290178 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.290083 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 03:00:45.463096 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.463053 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfm4" event={"ID":"de6141f3-f144-46ab-86c0-d780fcb0129e","Type":"ContainerStarted","Data":"8b46d4afbfdeebe7905690586b7823673b88b8c88f6f32631315e5c5f977f1bf"} Nov 03 03:00:45.464922 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.464887 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vg788" event={"ID":"162d9031-a839-4af7-8ba4-973b900c618e","Type":"ContainerStarted","Data":"eb6759ee7756a4a4eae6e725dad4951affd5ad463d7bf68627d3304e4088c7e3"} Nov 03 03:00:45.699080 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:45.699026 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4gfm4" podStartSLOduration=4.968772056 podStartE2EDuration="13.699010047s" totalImagesPullingTime="8.730237991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:45.579124368 +0000 UTC m=+14.758166739" watchObservedRunningTime="2026-11-03 03:00:45.699010047 +0000 UTC m=+14.878052418" Nov 03 03:00:46.344518 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.344490 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:46.344642 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.344576 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:46.344642 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.344590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:46.344713 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:46.344672 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:46.344768 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:46.344735 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:46.344811 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:46.344783 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:46.468286 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.468254 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="e10aaf933a32822625405fae1f94c75c79ab35f29869f2c122f53f39a7f94933" exitCode=0 Nov 03 03:00:46.468905 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.468340 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"e10aaf933a32822625405fae1f94c75c79ab35f29869f2c122f53f39a7f94933"} Nov 03 03:00:46.470506 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.470483 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" event={"ID":"ee1e4336-ab3b-4e70-a11c-8566fbc03510","Type":"ContainerStarted","Data":"f9c406f47850e588d97656da734cb940fc283a70d200a9722e646613d4235d2f"} Nov 03 03:00:46.690689 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.690581 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vg788" podStartSLOduration=4.207046578 podStartE2EDuration="14.690568227s" totalImagesPullingTime="10.483521649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:45.700720584 +0000 UTC m=+14.879762955" watchObservedRunningTime="2026-11-03 03:00:46.690568227 +0000 UTC m=+15.869610656" Nov 03 03:00:46.793414 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:46.793362 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" podStartSLOduration=3.61268774 podStartE2EDuration="14.793347396s" totalImagesPullingTime="11.180659656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:46.792963139 +0000 UTC m=+15.972005511" watchObservedRunningTime="2026-11-03 03:00:46.793347396 +0000 UTC m=+15.972389766" Nov 03 03:00:48.344409 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:48.344191 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:48.345018 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:48.344202 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:48.345018 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:48.344515 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:48.345018 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:48.344260 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:48.345018 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:48.344581 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:48.345018 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:48.344651 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:48.476649 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:48.476616 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="7616291c30875b2efbe3abde2f907166ececc0cb531bd068dc3079c0e436a096" exitCode=0 Nov 03 03:00:48.476795 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:48.476659 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"7616291c30875b2efbe3abde2f907166ececc0cb531bd068dc3079c0e436a096"} Nov 03 03:00:49.884410 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:49.884369 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:49.884882 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:49.884524 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 03:00:49.885196 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:49.885170 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:00:50.200455 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.200365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:50.200455 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.200410 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:50.200690 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.200526 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:50.200690 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.200527 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:50.200690 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.200579 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.200566518 +0000 UTC m=+35.379608868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:50.200690 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.200593 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.200587088 +0000 UTC m=+35.379629438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:50.301076 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.301045 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:50.301243 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.301223 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:50.301289 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.301245 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:50.301289 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.301254 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:50.301404 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.301311 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.301294899 +0000 UTC m=+35.480337251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:50.345251 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.345212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:50.345430 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.345212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:50.345430 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:50.345355 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:50.345430 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.345351 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:50.345579 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.345431 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:50.345579 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:50.345469 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:51.485531 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:51.485497 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="2f7b421751416be35d03fc82946970875b1b409bfab6886477bf4bb50a32b063" exitCode=0 Nov 03 03:00:51.485965 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:51.485563 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"2f7b421751416be35d03fc82946970875b1b409bfab6886477bf4bb50a32b063"} Nov 03 03:00:52.344660 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:52.344618 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:52.344814 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:52.344677 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:52.344814 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:52.344781 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:52.344814 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:52.344783 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:52.344969 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:52.344895 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:52.345007 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:52.344973 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:52.491656 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:52.491614 2571 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="c8fd460ffd92bdc0566b3c0382e821051e87799692dcd2af7578151ecdbb36ef" exitCode=0 Nov 03 03:00:52.492042 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:52.491673 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerDied","Data":"c8fd460ffd92bdc0566b3c0382e821051e87799692dcd2af7578151ecdbb36ef"} Nov 03 03:00:53.497720 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:53.497688 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" event={"ID":"4794a03d-e0ff-401e-b267-70361f49b0a0","Type":"ContainerStarted","Data":"d26da146e97ba5c5ddbbb2311566e5d18dd6db3055f2d91507ecbc6f43c623e9"} Nov 03 03:00:54.344528 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:54.344482 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:54.344528 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:54.344523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:54.344884 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:54.344499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:54.344884 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:54.344599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:54.344884 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:54.344655 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:54.344884 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:54.344729 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:56.344635 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:56.344587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:56.344635 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:56.344613 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:56.345194 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:56.344726 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:56.345194 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:56.344772 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:56.345194 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:56.344897 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:56.345194 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:56.344963 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:58.344630 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:58.344590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:00:58.344630 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:58.344622 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:00:58.344630 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:58.344625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:00:58.345121 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:58.344723 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:00:58.345121 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:58.344823 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:00:58.345121 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:00:58.344898 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:00:58.938678 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:58.938645 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ncxp7_fce981b0-12d5-4738-8dfe-2683c56f1a07/dns-node-resolver/0.log" Nov 03 03:00:59.563303 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:00:59.563263 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tdbsm_bf5ce847-feef-4cdf-b403-18eaa916c05c/node-ca/0.log" Nov 03 03:01:00.345058 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:00.345016 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:00.345058 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:00.345044 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:00.345304 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:00.345032 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:00.345304 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:00.345159 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:00.345304 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:00.345237 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:00.345428 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:00.345299 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:02.344732 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:02.344693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:02.345176 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:02.344693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:02.345176 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:02.344901 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:02.345176 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:02.344797 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:02.345176 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:02.344693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:02.345176 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:02.344994 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:03.522988 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.522806 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523327 2571 generic.go:356] "Generic (PLEG): container finished" podID="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" containerID="225c6d7deef9a8da3b05acfe8f542a658b6eac56e4fd9838b4dd38499eb22b72" exitCode=1 Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523374 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"fc7ce3e7939a7ee636b0db800a80b7f68576efc3dd1ac4f2d8c567de5601a649"} Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523429 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"150c74ab0f47117f849c2a3e1723a2dbaa5cb9717fbfd39a195ca5b017458482"} Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523461 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"000f80e04a553e540c698fcb83b97fb273a8befd2ccc661b91d87351bc285395"} Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523487 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"315c3aaa3affa34e6ca4f49f4a0d73be681fd6e8e212d6e5750b87bee5183ea1"} Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523513 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerDied","Data":"225c6d7deef9a8da3b05acfe8f542a658b6eac56e4fd9838b4dd38499eb22b72"} Nov 03 03:01:03.523562 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:03.523540 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"553ed60a3fa8208a64d851df683cf3fcc15069d1cf3873577bf9739221b67a9b"} Nov 03 03:01:04.345240 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:04.345206 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:04.345240 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:04.345234 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:04.345517 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:04.345313 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:04.345517 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:04.345363 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:04.345517 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:04.345437 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:04.345517 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:04.345484 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:05.530500 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:05.530473 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:05.531011 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:05.530792 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"07d50218764f7bec69cb0b2f6101a724a125c7d024e159a6418bee2eb49ac6b5"} Nov 03 03:01:06.211655 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.211615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:06.211655 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.211657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:06.211907 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.211762 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:01:06.211907 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.211770 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:01:06.211907 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.211814 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret podName:ba38cb3f-dfe8-4af7-b185-943337c4759e nodeName:}" failed. No retries permitted until 2026-11-03 03:01:38.211801845 +0000 UTC m=+67.390844194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret") pod "global-pull-secret-syncer-4vzrm" (UID: "ba38cb3f-dfe8-4af7-b185-943337c4759e") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:01:06.211907 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.211832 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs podName:7d221a1f-e9e2-4904-a5cb-55350a2c60bd nodeName:}" failed. No retries permitted until 2026-11-03 03:01:38.211818977 +0000 UTC m=+67.390861330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs") pod "network-metrics-daemon-4c62k" (UID: "7d221a1f-e9e2-4904-a5cb-55350a2c60bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:01:06.312411 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.312372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:06.312595 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.312510 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:01:06.312595 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.312529 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:01:06.312595 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.312542 2571 projected.go:196] Error preparing data for projected volume kube-api-access-55j87 for pod openshift-network-diagnostics/network-check-target-jk4nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:01:06.312595 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.312595 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87 podName:83cd9057-d746-4762-ba16-6bf9c83db135 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:38.312582058 +0000 UTC m=+67.491624412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-55j87" (UniqueName: "kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87") pod "network-check-target-jk4nq" (UID: "83cd9057-d746-4762-ba16-6bf9c83db135") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:01:06.344649 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.344620 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:06.344746 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.344693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:06.344810 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:06.344783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:06.344810 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.344792 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:06.344939 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.344891 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:06.344983 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:06.344967 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:07.538598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.538422 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:07.539232 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.538971 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"e325bf3ba44bb71974e4b79138b1c3f28ac802699114605abdfbffb1a8bf4b50"} Nov 03 03:01:07.539305 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.539279 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:07.539392 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.539375 2571 scope.go:122] "RemoveContainer" containerID="225c6d7deef9a8da3b05acfe8f542a658b6eac56e4fd9838b4dd38499eb22b72" Nov 03 03:01:07.558020 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.557999 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:07.820952 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:07.820835 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qwcqm" podStartSLOduration=23.374450723 podStartE2EDuration="35.820820588s" totalImagesPullingTime="12.446369865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:53.726922627 +0000 UTC m=+22.905964997" watchObservedRunningTime="2026-11-03 03:01:07.820820588 +0000 UTC m=+36.999862959" Nov 03 03:01:08.344970 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.344932 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:08.345154 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.345043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:08.345154 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:08.345133 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:08.345154 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:08.345049 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:08.345154 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.345053 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:08.345311 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:08.345206 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:08.544556 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.544524 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:08.544986 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.544874 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" event={"ID":"a75ca389-0c41-4d4f-9788-da6f8bb2cc70","Type":"ContainerStarted","Data":"5cce7b71331cf99c0cbd449b5a41f9393a56f75f1473ca0f13e5be86d2836a1e"} Nov 03 03:01:08.545155 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.545132 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:08.545290 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.545265 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:08.560453 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.560429 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:08.727152 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:08.727043 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" podStartSLOduration=8.953850032 podStartE2EDuration="36.727029434s" totalImagesPullingTime="27.773179402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:08.727004554 +0000 UTC m=+37.906046926" watchObservedRunningTime="2026-11-03 03:01:08.727029434 +0000 UTC m=+37.906071797" Nov 03 03:01:10.345183 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:10.345146 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:10.345183 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:10.345164 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:10.345653 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:10.345157 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:10.345653 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:10.345250 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:10.345653 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:10.345328 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:10.345653 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:10.345404 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:12.344991 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:12.344951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:12.344991 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:12.344986 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:12.345461 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:12.344958 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:12.345461 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:12.345063 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:12.345461 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:12.345150 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:12.345461 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:12.345265 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:14.344250 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:14.344200 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:14.344250 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:14.344257 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:14.344780 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:14.344217 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:14.344780 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:14.344339 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:14.344780 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:14.344391 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:14.344780 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:14.344479 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:16.345283 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:16.345239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:16.345706 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:16.345289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:16.345706 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:16.345343 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:16.345706 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:16.345352 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:16.345706 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:16.345406 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:16.345706 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:16.345510 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:18.344946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:18.344913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:18.344946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:18.344951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:18.345396 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:18.344951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:18.345396 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:18.345005 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:18.345396 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:18.345095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:18.345396 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:18.345148 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:20.344382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:20.344340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:20.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:20.344459 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:20.344838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:20.344351 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:20.344838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:20.344365 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:20.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:20.344542 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:20.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:20.344593 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:22.344377 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:22.344325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:22.344377 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:22.344376 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:22.344838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:22.344435 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:22.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:22.344450 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:22.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:22.344510 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:22.344838 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:22.344563 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:24.344420 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:24.344387 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:24.344946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:24.344444 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:24.344946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:24.344463 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:24.344946 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:24.344516 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:24.344946 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:24.344577 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:24.344946 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:24.344652 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:25.019359 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.019309 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vzrm"] Nov 03 03:01:25.019534 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.019443 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:25.019603 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:25.019554 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:25.020114 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.020076 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jk4nq"] Nov 03 03:01:25.020230 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.020166 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:25.020277 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:25.020257 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:25.025386 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.025350 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4c62k"] Nov 03 03:01:25.025515 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:25.025461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:25.025574 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:25.025551 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:26.345048 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:26.344963 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:26.345048 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:26.345014 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:26.345460 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:26.345082 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:26.345460 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:26.345205 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:27.345076 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:27.345043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:27.345506 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:27.345178 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jk4nq" podUID="83cd9057-d746-4762-ba16-6bf9c83db135" Nov 03 03:01:28.344937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:28.344724 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:28.345137 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:28.344779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:28.345137 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:28.345045 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4c62k" podUID="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" Nov 03 03:01:28.345137 ip-10-0-136-7 kubenswrapper[2571]: E1103 03:01:28.345078 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vzrm" podUID="ba38cb3f-dfe8-4af7-b185-943337c4759e" Nov 03 03:01:29.114165 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.114096 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeReady" Nov 03 03:01:29.114290 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.114221 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 03:01:29.237416 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.237382 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thwdd"] Nov 03 03:01:29.239088 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.239074 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.265655 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.265622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.265793 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.265677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jkl\" (UniqueName: \"kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.283787 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.283760 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mtw4b"] Nov 03 03:01:29.285536 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.285521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.296300 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.296266 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 03:01:29.296300 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.296269 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 03:01:29.296300 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.296304 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 03:01:29.296488 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.296358 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\"" Nov 03 03:01:29.301525 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.301503 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thwdd"] Nov 03 03:01:29.323029 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.322991 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtw4b"] Nov 03 03:01:29.340447 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.340414 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xv2gh"] Nov 03 03:01:29.342136 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.342120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.344335 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.344317 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:29.366999 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.366930 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.366999 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.366963 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.366999 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.366985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.367381 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.367381 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367243 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bf2n\" (UniqueName: \"kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.367381 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.367381 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jkl\" (UniqueName: \"kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.367381 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.367526 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.367526 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.367587 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.367531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbkj\" (UniqueName: \"kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.371949 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.371923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.400907 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.400873 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 03:01:29.400907 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.400873 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 03:01:29.401121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.400943 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\"" Nov 03 03:01:29.401121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.400997 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 03:01:29.401121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.401091 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 03:01:29.403410 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.403393 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 03:01:29.403491 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.403409 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\"" Nov 03 03:01:29.403491 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.403394 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\"" Nov 03 03:01:29.403706 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.403690 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 03:01:29.405027 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.405012 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 03:01:29.405282 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.405270 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 03:01:29.416762 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.416736 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv2gh"] Nov 03 03:01:29.468284 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468284 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.468546 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468337 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf2n\" (UniqueName: \"kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.468546 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468449 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.468546 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468546 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.468753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468578 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbkj\" (UniqueName: \"kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.468753 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468670 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.469037 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.468876 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.469172 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.469151 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.469172 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.469161 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.470896 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.470869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.470999 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.470933 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.482505 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.482480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jkl\" (UniqueName: \"kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.548747 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.548718 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:29.556625 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.556609 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf2n\" (UniqueName: \"kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.559486 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.559458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbkj\" (UniqueName: \"kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.597321 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.597293 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:29.656179 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.655994 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:29.946994 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.944179 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thwdd"] Nov 03 03:01:29.948346 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:29.948321 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297990c5_51d0_4280_b974_bf6521efb03e.slice/crio-11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d WatchSource:0}: Error finding container 11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d: Status 404 returned error can't find the container with id 11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d Nov 03 03:01:29.990049 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:29.990015 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtw4b"] Nov 03 03:01:29.993372 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:29.993353 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14dd09e_7152_49db_9326_343dc9aaaebb.slice/crio-818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5 WatchSource:0}: Error finding container 818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5: Status 404 returned error can't find the container with id 818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5 Nov 03 03:01:30.040811 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.040644 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv2gh"] Nov 03 03:01:30.043620 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:30.043590 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e39601_3470_4b7c_831e_5ee64bfa61ad.slice/crio-89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002 WatchSource:0}: Error finding container 89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002: Status 404 returned error can't find the container with id 89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002 Nov 03 03:01:30.344589 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.344554 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:30.344814 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.344591 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:30.399081 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.399047 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\"" Nov 03 03:01:30.399727 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.399101 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\"" Nov 03 03:01:30.399727 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.399101 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 03:01:30.399727 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.399123 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 03:01:30.591982 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.591942 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thwdd" event={"ID":"297990c5-51d0-4280-b974-bf6521efb03e","Type":"ContainerStarted","Data":"11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d"} Nov 03 03:01:30.593067 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.593045 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv2gh" event={"ID":"53e39601-3470-4b7c-831e-5ee64bfa61ad","Type":"ContainerStarted","Data":"89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002"} Nov 03 03:01:30.595683 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.595298 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtw4b" event={"ID":"a14dd09e-7152-49db-9326-343dc9aaaebb","Type":"ContainerStarted","Data":"fbad63d4850dfe1712e7705c1a664c48f880e4b5941db8675171fbd0df5288b8"} Nov 03 03:01:30.595683 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:30.595349 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtw4b" event={"ID":"a14dd09e-7152-49db-9326-343dc9aaaebb","Type":"ContainerStarted","Data":"818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5"} Nov 03 03:01:32.602137 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:32.602102 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtw4b" event={"ID":"a14dd09e-7152-49db-9326-343dc9aaaebb","Type":"ContainerStarted","Data":"9c1265a70216218e21597c5dcfc3693651d9accd50ec5f3b4bb9b80de84019d8"} Nov 03 03:01:33.607893 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:33.607833 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv2gh" event={"ID":"53e39601-3470-4b7c-831e-5ee64bfa61ad","Type":"ContainerStarted","Data":"6a2a8b25769f6b61bf8df50f14cae61e0bd50938e3e6aedbdc84c8f496b153ea"} Nov 03 03:01:34.613935 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:34.613896 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv2gh" event={"ID":"53e39601-3470-4b7c-831e-5ee64bfa61ad","Type":"ContainerStarted","Data":"058cdd34455309f6d81d157ae353c22cf134c4aebdb7829070782bc30af78b7a"} Nov 03 03:01:34.614366 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:34.614060 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:34.736000 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:34.735937 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xv2gh" podStartSLOduration=2.311505047 podStartE2EDuration="5.735917579s" totalImagesPullingTime="3.424412532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:34.73534705 +0000 UTC m=+63.914389421" watchObservedRunningTime="2026-11-03 03:01:34.735917579 +0000 UTC m=+63.914959950" Nov 03 03:01:36.621317 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:36.621281 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtw4b" event={"ID":"a14dd09e-7152-49db-9326-343dc9aaaebb","Type":"ContainerStarted","Data":"f47de47556ee6be75d01b4ffcfcab6403b5bbe8d237eb46993970b39d231291b"} Nov 03 03:01:36.871099 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:36.871048 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mtw4b" podStartSLOduration=2.288632606 podStartE2EDuration="7.871033015s" totalImagesPullingTime="5.582400409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:36.823044365 +0000 UTC m=+66.002086751" watchObservedRunningTime="2026-11-03 03:01:36.871033015 +0000 UTC m=+66.050075385" Nov 03 03:01:37.583116 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.583085 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75"] Nov 03 03:01:37.602104 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.602071 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.631148 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.631120 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mgf2f"] Nov 03 03:01:37.644238 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.644206 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thwdd" event={"ID":"297990c5-51d0-4280-b974-bf6521efb03e","Type":"ContainerStarted","Data":"3a27f17406b908e591b75eff4f88e8c110f352227b02412e0c820439588cbf26"} Nov 03 03:01:37.644382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.644319 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.660673 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.660648 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75"] Nov 03 03:01:37.679227 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.679197 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 03:01:37.679724 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.679704 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-crpwp\"" Nov 03 03:01:37.679724 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.679716 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 03:01:37.680001 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.679988 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 03:01:37.680045 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.680034 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 03:01:37.710521 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.710491 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 03:01:37.710682 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.710539 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 03:01:37.734866 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.734812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.735030 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.734872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735030 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.734905 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.735030 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.734987 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735499 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.735634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735509 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735545 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.735634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735634 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735826 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.735942 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2tn\" (UniqueName: \"kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.735994 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.735955 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.736116 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.736099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvcb\" (UniqueName: \"kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.736151 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.736136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.736192 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.736160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.759680 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.759645 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ccc7x\"" Nov 03 03:01:37.760103 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.760086 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 03:01:37.760181 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.760086 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 03:01:37.772101 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.772075 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 03:01:37.836838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.836838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.836838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.836975 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837121 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2tn\" (UniqueName: \"kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvcb\" (UniqueName: \"kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837235 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837275 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837449 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837350 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837799 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837501 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837799 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837799 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.837799 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837784 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.838027 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.838027 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.837986 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.840861 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.840824 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.840976 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.840909 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.840976 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.840930 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.841154 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.841137 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.901838 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.901800 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvcb\" (UniqueName: \"kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:37.904750 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.904723 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2tn\" (UniqueName: \"kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.951167 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.951132 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:37.999811 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:37.999751 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thwdd" podStartSLOduration=1.711737914 podStartE2EDuration="8.999734312s" totalImagesPullingTime="7.287996398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:37.998656398 +0000 UTC m=+67.177698769" watchObservedRunningTime="2026-11-03 03:01:37.999734312 +0000 UTC m=+67.178776685" Nov 03 03:01:38.003441 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.003415 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:38.040662 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:38.040635 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f384fb_bf15_4268_a97f_e0d537873319.slice/crio-12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23 WatchSource:0}: Error finding container 12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23: Status 404 returned error can't find the container with id 12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23 Nov 03 03:01:38.241509 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.241425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:38.241509 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.241464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:38.293209 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.293180 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 03:01:38.293384 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.293291 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 03:01:38.304711 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.304680 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:38.304827 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.304683 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:38.342696 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.342642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:38.396301 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.396270 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 03:01:38.408028 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.407998 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75"] Nov 03 03:01:38.410749 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:38.410720 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07201023_0dbc_4723_a3d1_d055fa9be1dc.slice/crio-7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8 WatchSource:0}: Error finding container 7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8: Status 404 returned error can't find the container with id 7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8 Nov 03 03:01:38.456736 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.456705 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 03:01:38.466420 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.466390 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:38.507925 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.507833 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\"" Nov 03 03:01:38.513415 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.513395 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\"" Nov 03 03:01:38.516405 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.516390 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:38.522635 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.522615 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:38.640303 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.640265 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" event={"ID":"07201023-0dbc-4723-a3d1-d055fa9be1dc","Type":"ContainerStarted","Data":"7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8"} Nov 03 03:01:38.641589 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.641560 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgf2f" event={"ID":"f0f384fb-bf15-4268-a97f-e0d537873319","Type":"ContainerStarted","Data":"12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23"} Nov 03 03:01:38.709016 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.708983 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\"" Nov 03 03:01:38.718191 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.718163 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:38.927427 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.927398 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vzrm"] Nov 03 03:01:38.930659 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:38.930634 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba38cb3f_dfe8_4af7_b185_943337c4759e.slice/crio-549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390 WatchSource:0}: Error finding container 549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390: Status 404 returned error can't find the container with id 549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390 Nov 03 03:01:38.976989 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:38.976954 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4c62k"] Nov 03 03:01:38.979609 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:38.979573 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d221a1f_e9e2_4904_a5cb_55350a2c60bd.slice/crio-4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05 WatchSource:0}: Error finding container 4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05: Status 404 returned error can't find the container with id 4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05 Nov 03 03:01:39.135178 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:39.132931 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jk4nq"] Nov 03 03:01:39.138886 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:39.138832 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cd9057_d746_4762_ba16_6bf9c83db135.slice/crio-865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef WatchSource:0}: Error finding container 865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef: Status 404 returned error can't find the container with id 865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef Nov 03 03:01:39.645924 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:39.645877 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vzrm" event={"ID":"ba38cb3f-dfe8-4af7-b185-943337c4759e","Type":"ContainerStarted","Data":"549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390"} Nov 03 03:01:39.647141 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:39.647116 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jk4nq" event={"ID":"83cd9057-d746-4762-ba16-6bf9c83db135","Type":"ContainerStarted","Data":"865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef"} Nov 03 03:01:39.648189 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:39.648169 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4c62k" event={"ID":"7d221a1f-e9e2-4904-a5cb-55350a2c60bd","Type":"ContainerStarted","Data":"4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05"} Nov 03 03:01:40.435947 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.435920 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cb88f77ff-knhmq"] Nov 03 03:01:40.439030 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.439005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.459534 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.459509 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cb88f77ff-knhmq"] Nov 03 03:01:40.506652 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.506615 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 03:01:40.507615 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.507592 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-48h7s1kcmmk62\"" Nov 03 03:01:40.507615 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.507605 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 03:01:40.507834 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.507817 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 03:01:40.507992 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.507866 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dh9lv\"" Nov 03 03:01:40.508291 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.508274 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 03:01:40.525399 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.525370 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 03:01:40.559558 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559733 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559733 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559733 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559648 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knj2\" (UniqueName: \"kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559733 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559733 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559715 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559933 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559933 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.559933 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.559824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.564100 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.564075 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:40.654019 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.653983 2571 generic.go:356] "Generic (PLEG): container finished" podID="f0f384fb-bf15-4268-a97f-e0d537873319" containerID="efea0e4285dd0bd2b6a631e6bd4b1f1fd024e3c49aa2fefc2a020acd24c99233" exitCode=0 Nov 03 03:01:40.654407 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.654042 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgf2f" event={"ID":"f0f384fb-bf15-4268-a97f-e0d537873319","Type":"ContainerDied","Data":"efea0e4285dd0bd2b6a631e6bd4b1f1fd024e3c49aa2fefc2a020acd24c99233"} Nov 03 03:01:40.660665 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.660789 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660678 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.660789 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.660921 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.660921 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.660921 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.661069 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knj2\" (UniqueName: \"kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.661069 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.661069 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.660992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.662199 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.662151 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.663060 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.663028 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.664926 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.664879 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.665435 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.665392 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.666016 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.665997 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.667596 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.667555 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.667987 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.667968 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.668520 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.668496 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.728150 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.728126 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knj2\" (UniqueName: \"kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:40.771804 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:40.771776 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:41.229824 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.229795 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cb88f77ff-knhmq"] Nov 03 03:01:41.233165 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:41.233129 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cf23f3_5db2_49c6_9576_a824fa35da0c.slice/crio-1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31 WatchSource:0}: Error finding container 1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31: Status 404 returned error can't find the container with id 1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31 Nov 03 03:01:41.662946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.662094 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgf2f" event={"ID":"f0f384fb-bf15-4268-a97f-e0d537873319","Type":"ContainerStarted","Data":"11e44bdafdc13f16a0d56a9375bddfc6cf67bdacdf6a1b5a5f79bdb6043b590e"} Nov 03 03:01:41.662946 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.662162 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgf2f" event={"ID":"f0f384fb-bf15-4268-a97f-e0d537873319","Type":"ContainerStarted","Data":"8b47161c08465b956a6cbd60afef3cc4902e252f8c9564284fa4a41c1b3d282a"} Nov 03 03:01:41.668283 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.668174 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31"} Nov 03 03:01:41.671765 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.671736 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" event={"ID":"07201023-0dbc-4723-a3d1-d055fa9be1dc","Type":"ContainerStarted","Data":"b312d316b1e2d7cd72c37ab23f413d3db1405967d8885e4d6eb6c60a8c760a10"} Nov 03 03:01:41.671921 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.671797 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" event={"ID":"07201023-0dbc-4723-a3d1-d055fa9be1dc","Type":"ContainerStarted","Data":"4dd41eff895a32932694b3edfdf5f8170fce6dc82b1f60bb186a2eef33867176"} Nov 03 03:01:41.671921 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.671838 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" event={"ID":"07201023-0dbc-4723-a3d1-d055fa9be1dc","Type":"ContainerStarted","Data":"f4f66e3accf6c3205c94a16e955d911a3515f0d760f042304b814ef9f8b233fd"} Nov 03 03:01:41.674260 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.674226 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4c62k" event={"ID":"7d221a1f-e9e2-4904-a5cb-55350a2c60bd","Type":"ContainerStarted","Data":"fb9e84647aa075435c9fde3f35ddcb7852297e92e42babd20f4464b544ecf854"} Nov 03 03:01:41.928413 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:41.928313 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mgf2f" podStartSLOduration=2.465979475 podStartE2EDuration="4.928298729s" totalImagesPullingTime="2.371364961s" totalInitContainerRuntime="90.954293ms" isStatefulPod=true podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:41.795295638 +0000 UTC m=+70.974338010" watchObservedRunningTime="2026-11-03 03:01:41.928298729 +0000 UTC m=+71.107341100" Nov 03 03:01:42.080560 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.080498 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" podStartSLOduration=2.248136618 podStartE2EDuration="5.08048104s" totalImagesPullingTime="2.832344422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:41.928956189 +0000 UTC m=+71.107998561" watchObservedRunningTime="2026-11-03 03:01:42.08048104 +0000 UTC m=+71.259523411" Nov 03 03:01:42.080989 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.080972 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b"] Nov 03 03:01:42.083193 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.083171 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.148070 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.148031 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b"] Nov 03 03:01:42.162060 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.162025 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 03:01:42.162235 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.162174 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gdrqn\"" Nov 03 03:01:42.177299 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.177261 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdrq\" (UniqueName: \"kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.177299 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.177308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.278039 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.277999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdrq\" (UniqueName: \"kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.278039 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.278050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.280834 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.280802 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.347642 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.347612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdrq\" (UniqueName: \"kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.394269 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.394225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:42.683178 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.683009 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4c62k" event={"ID":"7d221a1f-e9e2-4904-a5cb-55350a2c60bd","Type":"ContainerStarted","Data":"ab32b66aee55918e938258fbbd4051637e3bb92c09654c50e0dfa586b4553515"} Nov 03 03:01:42.784766 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:42.784692 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b"] Nov 03 03:01:42.789636 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:42.789604 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eb8632_4e51_4b33_bb06_fb6b3fbfd6ad.slice/crio-62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9 WatchSource:0}: Error finding container 62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9: Status 404 returned error can't find the container with id 62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9 Nov 03 03:01:43.688178 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:43.688139 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" event={"ID":"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad","Type":"ContainerStarted","Data":"62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9"} Nov 03 03:01:44.622133 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:44.622102 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:44.755181 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:44.755125 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4c62k" podStartSLOduration=70.282300034 podStartE2EDuration="1m12.755108021s" totalImagesPullingTime="2.472807987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:42.804780785 +0000 UTC m=+71.983823156" watchObservedRunningTime="2026-11-03 03:01:44.755108021 +0000 UTC m=+73.934150395" Nov 03 03:01:45.718092 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:45.717487 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jk4nq" event={"ID":"83cd9057-d746-4762-ba16-6bf9c83db135","Type":"ContainerStarted","Data":"a568f2ffe96428cd1a9f50906a937530228c3626921ca342f171552b5eae2c3b"} Nov 03 03:01:45.718092 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:45.718042 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:45.720891 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:45.720779 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"9da55873a7d11dfe9350ceb5ec91f8eafce8d930c2da32a94b1e67bef3d25717"} Nov 03 03:01:46.727666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:46.727424 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"0973af380d63d96192796da7a1998394a4e444cd94bfdc0cccdf4f21f848e2c5"} Nov 03 03:01:46.727666 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:46.727609 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"cf40e9704e04679fa74fb4278686e8cd93ea621a789d383b98009c2a34ab2025"} Nov 03 03:01:46.729424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:46.729397 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" event={"ID":"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad","Type":"ContainerStarted","Data":"0fcc82502f479e032c15bfa6aeaf61ce061eed79bb8063d89018b611e3fd180f"} Nov 03 03:01:46.854058 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:46.853997 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jk4nq" podStartSLOduration=68.396450245 podStartE2EDuration="1m14.853977777s" totalImagesPullingTime="6.457527532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:45.838064415 +0000 UTC m=+75.017106788" watchObservedRunningTime="2026-11-03 03:01:46.853977777 +0000 UTC m=+76.033020160" Nov 03 03:01:46.855034 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:46.854994 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" podStartSLOduration=1.6110578960000002 podStartE2EDuration="4.854977739s" totalImagesPullingTime="3.243919843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:46.852932479 +0000 UTC m=+76.031974851" watchObservedRunningTime="2026-11-03 03:01:46.854977739 +0000 UTC m=+76.034020111" Nov 03 03:01:47.304751 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.304715 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:01:47.307632 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.307611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.363333 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.363293 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 03:01:47.363509 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.363395 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 03:01:47.430764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.430764 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431017 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431017 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431017 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431017 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.430960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431218 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431218 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431151 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431218 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431313 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431313 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431382 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431464 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431390 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431464 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431464 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431563 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431563 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431619 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431696 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431627 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.431696 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.431655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.463083 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463051 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 03:01:47.463083 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463073 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 03:01:47.463331 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463185 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 03:01:47.463331 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463185 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 03:01:47.463331 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463272 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 03:01:47.463438 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463358 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 03:01:47.463438 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463406 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 03:01:47.463506 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463463 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 03:01:47.463574 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.463548 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 03:01:47.465484 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.465464 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:01:47.513584 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513545 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 03:01:47.513788 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513660 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ba8kqj028vm5e\"" Nov 03 03:01:47.513788 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513730 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 03:01:47.513788 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513559 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 03:01:47.513975 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513805 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 03:01:47.513975 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.513906 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cn4p\"" Nov 03 03:01:47.532424 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532598 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532613 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532698 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532723 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.532839 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.533247 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.533247 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532933 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.533247 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.532960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.534717 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.534126 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.535281 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.535250 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.535900 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.535873 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.536868 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.536827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.538415 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.538367 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.538515 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.538499 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.540294 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.540270 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.540675 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.540651 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.541077 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.541055 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.541728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.542297 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.542451 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.542898 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.543015 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.543601 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.543901 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.544021 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.544937 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.544241 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.548891 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.548869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.604303 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.604230 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.627572 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.627541 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:47.735875 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.735823 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:47.742300 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:47.742276 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:48.076073 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.076030 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:01:48.079356 ip-10-0-136-7 kubenswrapper[2571]: W1103 03:01:48.079324 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f110ed_1cbf_478a_a45f_c2181a1a5563.slice/crio-d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a WatchSource:0}: Error finding container d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a: Status 404 returned error can't find the container with id d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a Nov 03 03:01:48.739994 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.739959 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vzrm" event={"ID":"ba38cb3f-dfe8-4af7-b185-943337c4759e","Type":"ContainerStarted","Data":"f9aa34384050ea5cdeb038306af1d5abb57af07e1dcef960de7c8f778daae2f1"} Nov 03 03:01:48.743403 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.743376 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"e8578ad8f1b5855e7001a89e2b75d9c6fb0eb077c0d3cf2195c45b6810b2db0a"} Nov 03 03:01:48.743534 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.743441 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"5fd1e8a653de22e46ccba407eb40b0be6fc23a9fb662673dfae0f192a97a43a0"} Nov 03 03:01:48.743534 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.743478 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" event={"ID":"92cf23f3-5db2-49c6-9576-a824fa35da0c","Type":"ContainerStarted","Data":"8a2a2c3d9db90f778b6cfcab50cc0bba9dc3e065856a7f7f721e27149f58f65d"} Nov 03 03:01:48.743534 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.743521 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:48.744612 ip-10-0-136-7 kubenswrapper[2571]: I1103 03:01:48.744589 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a"} Nov 03 03:01:48.790517 ip-10-0-136-7 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 03:01:48.802285 ip-10-0-136-7 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 03:01:48.802461 ip-10-0-136-7 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 03:01:48.802641 ip-10-0-136-7 systemd[1]: kubelet.service: Consumed 4.062s CPU time. Nov 03 03:01:48.812408 ip-10-0-136-7 systemd[1]: Starting Kubernetes Kubelet... Nov 03 03:01:48.876353 ip-10-0-136-7 kubenswrapper[5275]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:48.876353 ip-10-0-136-7 kubenswrapper[5275]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 03:01:48.876353 ip-10-0-136-7 kubenswrapper[5275]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:48.876353 ip-10-0-136-7 kubenswrapper[5275]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:48.878609 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878524 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.878609 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878610 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878615 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878618 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878621 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878625 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878629 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878632 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878635 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878637 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878640 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878643 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878645 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878648 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878650 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878653 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878656 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878658 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878660 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878663 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878666 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.878677 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878668 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878671 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878674 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878678 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878680 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878684 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878687 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878690 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878693 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878695 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878698 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878702 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878707 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878710 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878713 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878716 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878719 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878722 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.879215 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878725 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878727 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878730 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878733 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878735 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878738 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878741 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878744 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878747 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878750 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878753 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878756 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878758 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878761 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878764 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878767 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878769 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878773 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878777 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.879640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878779 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878782 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878795 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878797 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878800 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878803 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878805 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878808 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878811 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878813 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878816 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878819 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878822 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878825 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878828 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878831 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878833 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878836 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878839 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878856 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.880353 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878858 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878861 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878864 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878866 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878869 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878872 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878874 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878877 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878880 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878882 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878885 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878889 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878892 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878894 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878897 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878899 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878902 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878904 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878907 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878909 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.881184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878912 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878914 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878917 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878919 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878922 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878924 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878927 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.878930 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880134 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880151 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880157 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880161 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880166 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880171 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880175 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880179 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880189 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880193 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880197 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880201 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.881763 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880206 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880210 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880215 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880220 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880225 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880229 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880233 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880237 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880247 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880251 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880255 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880259 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880263 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880269 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880273 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880277 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880281 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880285 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880289 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880293 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.882456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880299 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880308 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880312 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880316 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880320 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880324 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880329 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880333 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880337 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880342 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880347 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880351 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880356 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880360 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880370 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880374 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880378 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880383 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880387 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.883005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880392 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880396 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880401 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880405 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880409 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880415 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880419 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880423 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880432 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880436 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880440 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880445 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880449 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880453 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880460 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880464 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880469 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880476 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880482 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.883453 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880492 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880497 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880501 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880505 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880509 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880514 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880518 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880523 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880527 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880531 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880535 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880540 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880544 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880554 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880558 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880562 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880567 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880571 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880575 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.883933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880580 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880584 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880588 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880594 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880598 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880602 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880611 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880620 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880627 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880652 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880658 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880661 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880664 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880667 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880670 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880672 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.880675 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882196 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882208 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.884427 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882212 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882217 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882223 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882226 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882229 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882232 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882235 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882238 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882241 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882243 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882246 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882249 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882252 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882254 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882257 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882260 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882262 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882265 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882267 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.884919 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882270 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882273 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882275 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882278 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882281 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882284 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882287 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882290 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882293 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882296 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882299 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882302 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882305 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882307 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882310 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882313 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882317 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882321 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882324 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882328 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.885412 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882331 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882334 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882337 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882340 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882343 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882345 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882348 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882350 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882353 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882355 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882358 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882361 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882363 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882366 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882369 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882372 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882375 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882377 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882380 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882384 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.885904 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882387 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882389 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882392 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882395 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882398 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882401 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882403 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882406 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882409 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882411 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882414 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882417 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882419 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882422 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882426 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882429 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882432 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882434 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882437 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882440 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.886391 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882442 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882445 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882449 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882451 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882454 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882457 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882459 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882462 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882465 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882470 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882473 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882476 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882479 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882482 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882484 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882487 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882490 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882492 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882495 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.886888 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882498 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882500 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882503 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882505 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882508 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.882511 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882608 5275 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882617 5275 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882624 5275 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882629 5275 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882633 5275 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882636 5275 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882641 5275 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882647 5275 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882651 5275 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882654 5275 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882657 5275 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882661 5275 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882664 5275 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882667 5275 flags.go:64] FLAG: --cgroup-root="" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882669 5275 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882672 5275 flags.go:64] FLAG: --client-ca-file="" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882676 5275 flags.go:64] FLAG: --cloud-provider="external" Nov 03 03:01:48.887356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882679 5275 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882684 5275 flags.go:64] FLAG: --cluster-domain="" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882687 5275 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882690 5275 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882693 5275 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882696 5275 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882700 5275 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882703 5275 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882706 5275 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882709 5275 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882712 5275 flags.go:64] FLAG: --contention-profiling="false" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882715 5275 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882718 5275 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882721 5275 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882724 5275 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882728 5275 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882731 5275 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882734 5275 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882738 5275 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882741 5275 flags.go:64] FLAG: --enable-server="true" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882744 5275 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882748 5275 flags.go:64] FLAG: --event-burst="100" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882753 5275 flags.go:64] FLAG: --event-qps="50" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882756 5275 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 03:01:48.887915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882759 5275 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882762 5275 flags.go:64] FLAG: --eviction-hard="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882767 5275 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882770 5275 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882773 5275 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882777 5275 flags.go:64] FLAG: --eviction-soft="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882780 5275 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882782 5275 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882785 5275 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882788 5275 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882791 5275 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882794 5275 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882796 5275 flags.go:64] FLAG: --feature-gates="" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882801 5275 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882804 5275 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882807 5275 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882810 5275 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882813 5275 flags.go:64] FLAG: --healthz-port="10248" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882816 5275 flags.go:64] FLAG: --help="false" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882819 5275 flags.go:64] FLAG: --hostname-override="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882822 5275 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882826 5275 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882828 5275 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 03:01:48.888480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882832 5275 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882835 5275 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882838 5275 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882841 5275 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882859 5275 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882863 5275 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882866 5275 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882870 5275 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882874 5275 flags.go:64] FLAG: --kube-reserved="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882877 5275 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882879 5275 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882883 5275 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882886 5275 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882888 5275 flags.go:64] FLAG: --lock-file="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882891 5275 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882894 5275 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882897 5275 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882902 5275 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882905 5275 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882908 5275 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882911 5275 flags.go:64] FLAG: --logging-format="text" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882914 5275 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882917 5275 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882920 5275 flags.go:64] FLAG: --manifest-url="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882923 5275 flags.go:64] FLAG: --manifest-url-header="" Nov 03 03:01:48.889052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882928 5275 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882931 5275 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882935 5275 flags.go:64] FLAG: --max-pods="110" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882938 5275 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882941 5275 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882944 5275 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882947 5275 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882950 5275 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882952 5275 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882955 5275 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882962 5275 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882965 5275 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882968 5275 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882972 5275 flags.go:64] FLAG: --pod-cidr="" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882976 5275 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882978 5275 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882982 5275 flags.go:64] FLAG: --pods-per-core="0" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882984 5275 flags.go:64] FLAG: --port="10250" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882987 5275 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882990 5275 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d535352506dc27d7" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882993 5275 flags.go:64] FLAG: --qos-reserved="" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882996 5275 flags.go:64] FLAG: --read-only-port="10255" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.882999 5275 flags.go:64] FLAG: --register-node="true" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883002 5275 flags.go:64] FLAG: --register-with-taints="" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883005 5275 flags.go:64] FLAG: --registry-burst="10" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883008 5275 flags.go:64] FLAG: --registry-qps="5" Nov 03 03:01:48.889636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883011 5275 flags.go:64] FLAG: --reserved-cpus="" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883014 5275 flags.go:64] FLAG: --reserved-memory="" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883018 5275 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883021 5275 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883025 5275 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883027 5275 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883030 5275 flags.go:64] FLAG: --runonce="false" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883033 5275 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883037 5275 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883040 5275 flags.go:64] FLAG: --seccomp-default="false" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883043 5275 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883046 5275 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883049 5275 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883052 5275 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883055 5275 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883058 5275 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883061 5275 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883064 5275 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883067 5275 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883070 5275 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883075 5275 flags.go:64] FLAG: --system-cgroups="" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883078 5275 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883084 5275 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883087 5275 flags.go:64] FLAG: --tls-cert-file="" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883090 5275 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 03:01:48.890317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883094 5275 flags.go:64] FLAG: --tls-min-version="" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883097 5275 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883099 5275 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883102 5275 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883105 5275 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883108 5275 flags.go:64] FLAG: --v="2" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883112 5275 flags.go:64] FLAG: --version="false" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883116 5275 flags.go:64] FLAG: --vmodule="" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883120 5275 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883123 5275 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883268 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883272 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883275 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883278 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883281 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883285 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883288 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883291 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883293 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883296 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883299 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883301 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.890951 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883304 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883306 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883309 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883311 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883314 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883318 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883321 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883324 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883326 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883329 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883331 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883334 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883337 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883340 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883342 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883345 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883347 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883350 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883353 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883356 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.891488 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883358 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883361 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883364 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883367 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883369 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883373 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883376 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883379 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883382 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883385 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883387 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883390 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883392 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883395 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883398 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883401 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883403 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883407 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883411 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.892005 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883414 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883416 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883419 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883421 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883424 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883427 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883429 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883432 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883434 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883437 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883440 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883442 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883445 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883448 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883450 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883452 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883455 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883458 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883462 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.892868 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883465 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883468 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883470 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883474 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883476 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883479 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883482 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883485 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883487 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883490 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883492 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883496 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883498 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883501 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883504 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883507 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883509 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883511 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883514 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883517 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.893370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883519 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883522 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883525 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883529 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883533 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883536 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883539 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883541 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883544 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883546 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883549 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883554 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883557 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883561 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883563 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.893885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883566 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.883592 5275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883635 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883639 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883642 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883645 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883649 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883653 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883657 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883660 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883663 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883666 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883676 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883680 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883683 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.894393 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883685 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883688 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883691 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883693 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883696 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883699 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883701 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883704 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883706 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883709 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883711 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883714 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883716 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883719 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883723 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883726 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883728 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883731 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883734 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883737 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.894772 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883739 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883742 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883744 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883747 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883749 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883754 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883756 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883759 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883762 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883765 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883768 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883771 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883773 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883776 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883778 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883781 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883783 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883786 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883788 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883791 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.895404 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883793 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883796 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883798 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883801 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883803 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883806 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883810 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883813 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883816 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883818 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883821 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883823 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883826 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883829 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883831 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883834 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883836 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883840 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883858 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.896010 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883861 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883864 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883866 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883870 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883873 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883875 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883878 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883880 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883883 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883886 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883889 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883892 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883895 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883897 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883900 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883903 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883905 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883908 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883910 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883914 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.896649 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883917 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883920 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883922 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883925 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883927 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883930 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883932 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883935 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883938 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883941 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883946 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883950 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883954 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.883957 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.887881 5275 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.887896 5275 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.887914 5275 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.887918 5275 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887972 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.897227 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887979 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887984 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887987 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887990 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887992 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887995 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.887998 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888000 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888003 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888006 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888008 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888011 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888014 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888017 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888020 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888022 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888025 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888028 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888030 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.897676 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888033 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888035 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888038 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888040 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888043 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888046 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888049 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888051 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888054 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888056 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888059 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888062 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888064 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888067 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888069 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888072 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888074 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888077 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888079 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888082 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.898146 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888084 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888087 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888089 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888092 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888094 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888097 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888100 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888103 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888105 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888108 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888110 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888113 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888116 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888118 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888121 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888123 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888126 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888129 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888131 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.898616 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888134 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888137 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888141 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888145 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888149 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888151 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888154 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888156 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888159 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888162 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888164 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888167 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888169 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888172 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888174 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888177 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888179 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888182 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.899301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888185 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888187 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888190 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888193 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888195 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888198 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888201 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888203 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888206 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888208 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888210 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888213 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888216 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888218 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888221 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888223 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888226 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888228 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888230 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888234 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.900025 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888236 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888239 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888241 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888244 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888246 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888249 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888252 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888254 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888257 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.888284 5275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888420 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888425 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888428 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888430 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888433 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888436 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.900526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888438 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888441 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888443 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888446 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888449 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888451 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888454 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888456 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888459 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888461 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888464 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888467 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888469 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888472 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888474 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888480 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888484 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888486 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888490 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888492 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.901070 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888495 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888498 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888501 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888503 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888506 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888508 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888511 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888514 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888516 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888519 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888522 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888524 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888527 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888529 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888532 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888534 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888537 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888539 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888542 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888545 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.901549 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888547 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888550 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888552 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888556 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888560 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888563 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888565 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888569 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888572 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888575 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888578 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888581 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888584 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888587 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888589 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888592 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888594 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888597 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888599 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.902041 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888602 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888605 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888607 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888610 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888613 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888615 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888618 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888621 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888623 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888626 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888628 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888631 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888633 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888636 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888639 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888641 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888644 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888647 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888649 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888652 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.902524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888656 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888658 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888661 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888665 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888668 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888670 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888673 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888676 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888680 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888683 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888687 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888691 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888694 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888699 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888701 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888704 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888707 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888709 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888712 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.903122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888714 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888717 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.888739 5275 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888777 5275 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888781 5275 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888784 5275 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888787 5275 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888789 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888792 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888795 5275 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888797 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888799 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888802 5275 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888804 5275 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888809 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888811 5275 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:48.903624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888814 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888825 5275 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888828 5275 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888831 5275 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888833 5275 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888837 5275 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888840 5275 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888863 5275 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888868 5275 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888872 5275 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888876 5275 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888878 5275 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888881 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888884 5275 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888887 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888889 5275 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888892 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888895 5275 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888900 5275 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888903 5275 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:48.904042 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888906 5275 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888909 5275 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888912 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888914 5275 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888918 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888920 5275 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888923 5275 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888925 5275 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888928 5275 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888930 5275 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888936 5275 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888938 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888941 5275 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888943 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888947 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888950 5275 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888953 5275 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888957 5275 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888959 5275 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888962 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:48.904507 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888964 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888967 5275 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888970 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888972 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888974 5275 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888977 5275 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888980 5275 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888983 5275 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888985 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888987 5275 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888990 5275 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888992 5275 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888995 5275 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.888997 5275 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889000 5275 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889003 5275 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889005 5275 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889008 5275 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889010 5275 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889013 5275 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:48.905026 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889016 5275 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889020 5275 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889023 5275 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889028 5275 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889032 5275 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889035 5275 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889039 5275 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889047 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889050 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889054 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889056 5275 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889060 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889062 5275 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889066 5275 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889068 5275 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889071 5275 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889074 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889076 5275 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889079 5275 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:48.905561 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889081 5275 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889084 5275 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889086 5275 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889089 5275 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889091 5275 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889095 5275 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889098 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889100 5275 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889103 5275 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889105 5275 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889108 5275 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889110 5275 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889113 5275 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:01:48.889115 5275 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.889177 5275 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890193 5275 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890254 5275 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890440 5275 server.go:1014] "Starting client certificate rotation" Nov 03 03:01:48.906057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890576 5275 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890623 5275 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:31 +0000 UTC" deadline="2028-05-03 12:49:33.603390201 +0000 UTC" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890650 5275 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13137h47m44.712742175s" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.890921 5275 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.892113 5275 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.893738 5275 log.go:25] "Validated CRI v1 runtime API" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.895601 5275 log.go:25] "Validated CRI v1 image API" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.896353 5275 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 03:01:48.906489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.898518 5275 fs.go:135] Filesystem UUIDs: map[46cba2f0-ec2b-403e-8dd9-5313e7d4b767:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9160006a-86fd-4a30-8214-c03e07d39e64:/dev/nvme0n1p4] Nov 03 03:01:48.906717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.898540 5275 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~projected/kube-api-access-nl2tn:{mountpoint:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~projected/kube-api-access-nl2tn major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~projected/kube-api-access-97xxh:{mountpoint:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~projected/kube-api-access-97xxh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~projected/kube-api-access-g4jkl:{mountpoint:/var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~projected/kube-api-access-g4jkl major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~empty-dir/config-out major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/kube-api-access-v8bd8:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/kube-api-access-v8bd8 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/tls-assets major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/config major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/web-config major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4794a03d-e0ff-401e-b267-70361f49b0a0/volumes/kubernetes.io~projected/kube-api-access-258g4:{mountpoint:/var/lib/kubelet/pods/4794a03d-e0ff-401e-b267-70361f49b0a0/volumes/kubernetes.io~projected/kube-api-access-258g4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b95e773-2411-4fbc-b0f1-0e5fbde12efb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4b95e773-2411-4fbc-b0f1-0e5fbde12efb/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~projected/kube-api-access-5bf2n:{mountpoint:/var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~projected/kube-api-access-5bf2n major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~projected/kube-api-access-95g7q:{mountpoint:/var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~projected/kube-api-access-95g7q major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~secret/metrics-certs major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~projected/kube-api-access-7tdrq:{mountpoint:/var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~projected/kube-api-access-7tdrq major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83cd9057-d746-4762-ba16-6bf9c83db135/volumes/kubernetes.io~projected/kube-api-access-55j87:{mountpoint:/var/lib/kubelet/pods/83cd9057-d746-4762-ba16-6bf9c83db135/volumes/kubernetes.io~projected/kube-api-access-55j87 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~projected/kube-api-access-6knj2:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~projected/kube-api-access-6knj2 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~projected/kube-api-access-6pbkj:{mountpoint:/var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~projected/kube-api-access-6pbkj major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~projected/kube-api-access-kxxs5:{mountpoint:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~projected/kube-api-access-kxxs5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba38cb3f-dfe8-4af7-b185-943337c4759e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ba38cb3f-dfe8-4af7-b185-943337c4759e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf5ce847-feef-4cdf-b403-18eaa916c05c/volumes/kubernetes.io~projected/kube-api-access-ftswn:{mountpoint:/var/lib/kubelet/pods/bf5ce847-feef-4cdf-b403-18eaa916c05c/volumes/kubernetes.io~projected/kube-api-access-ftswn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6e95865-15f3-4a5d-9e0d-47a5c172ddde/volumes/kubernetes.io~projected/kube-api-access-nd9m9:{mountpoint:/var/lib/kubelet/pods/d6e95865-15f3-4a5d-9e0d-47a5c172ddde/volumes/kubernetes.io~projected/kube-api-access-nd9m9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6141f3-f144-46ab-86c0-d780fcb0129e/volumes/kubernetes.io~projected/kube-api-access-rffgl:{mountpoint:/var/lib/kubelet/pods/de6141f3-f144-46ab-86c0-d780fcb0129e/volumes/kubernetes.io~projected/kube-api-access-rffgl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee1e4336-ab3b-4e70-a11c-8566fbc03510/volumes/kubernetes.io~projected/kube-api-access-xjttr:{mountpoint:/var/lib/kubelet/pods/ee1e4336-ab3b-4e70-a11c-8566fbc03510/volumes/kubernetes.io~projected/kube-api-access-xjttr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~projected/kube-api-access-6pvcb:{mountpoint:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~projected/kube-api-access-6pvcb major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fce981b0-12d5-4738-8dfe-2683c56f1a07/volumes/kubernetes.io~projected/kube-api-access-z929b:{mountpoint:/var/lib/kubelet/pods/fce981b0-12d5-4738-8dfe-2683c56f1a07/volumes/kubernetes.io~projected/kube-api-access-z929b major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5b4c392aaff3fabf0e99dac94a3beada8e85bef29c91bfdf3636449b29a0645a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4090ff2900780a2e99257d29a00a8892c358c60f7cc9a864a40fa4b869a8d857/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f6460fec5023c1d3b200955126f06055b3d262333daa00c26295056a6efe43e0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/66b1934db95b63d5d94481d8eafb455cc0d5e490a6327179e8ff9ea6b2c93447/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c4f159451a986421496f58f499b297c453cb1ec14906dabc1529296a641cffe2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d5a0f6c4385ea8cd850dcc1edf3d6823ecdda8ad445492bd0e01e49a524c6f3c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/17616d5e93abb78584d4834dc2904c2eaa45265a3130bc975c55b68f58e26144/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dc2599c5c06f4541111fb5547ad0ffc945142f7a29f24ed23598a674001995ce/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b6f9005215c2315048295bdec4b62eeae1fdfca4ca75d22cf2b1ca56279f0191/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/90f1c63e0c46676b868bf925793656d1e9ae8b2f8fa90d89dd6a35909cb678e5/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/18f3dd1892d3eaf6c8eb9ec75106f47c641a4892b3ab7d02fcccc0728a1d5ce7/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/204b038761584cbd18e744f5b0937ed1fd5857130f0decfe92fa36cf4a59ba61/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fd3701416f787e524c8626e5f923e84c844c1ceb3c3e7a288c5b875d5af12c5b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/a1da6125f5ca5903bc1f61dd67cd24349f4b616baccea6c1c669b30cf2e230e5/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/2fab4e7d1c63a2226fbd18d72fa493a0c142241e683f872665d6dcbd7487e548/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/3b0b974ee2061dae4f5991702eeef9b985f431175f9b09dee7b39487f956d5ca/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/66e98869d604f5f2782f070ef8a2720795d8793669d13ef368115456b8eb1d53/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/673bfe1639e51b7f007eb0c964af087f265d7b99181904e98cde98aac374ed74/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/9b2a22cb7bb2222261d24a105674e36eb00677bf7b5dae2b660a8405d0a16e5a/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/e273a4681b39b37bfe3caf65bb7b8d2bb5ee65dc2e570fce4132416ecc1f19f9/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/2d42f6772f8122d5e83b6f862f0bf572f4ee15192bffda4723f454f060b42728/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/7935216fe097bcf0a8a88e699657c3e02e99dc24485a4b09c9f2640431c4e09c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0b0a22fc5c591ed3938ff6552d88dbfcccf67d29bf8cbd1906339c356ba4801a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9592cbd52438e60e15c13d565226b3f8630dc9152d3e9797b73ea3f4eb8655ea/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/b6cf44fb5d1bc5764cfc6e19716208714763faa2de30bc7f11160791ca3f784b/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/eeaf4e2555de65ce60c9c3118c8a4ab6052125de1db339df5d2c9b17b671cef5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/98a08f61d449b240a2c4dfc983e96cadbce6ef8ea8780d5a01d94e3211f9322c/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/7760d532cc3d631c7c3a067cf5b9e8ddb832c7cf8782aaa704d0c90452ada206/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/69f592d9d4fd6b0836d1673b2bc6655284b8345c9ef44209b2f1a3aabbb84f06/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8a8047d21ecae5060bf04d9dc89b08dc1f2eb4a5040a67664a5513261909440a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/6a7c8e40cdb907e4137db115a22023584f2d11ccc642863a8a7c574b4e1ff29f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/1398d7bfe7cafa7603c02617877859cbd40287e538a9586d92f69ff961e1508e/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/9a0621add535e2143a1e14af0c8853e4d72589613bc6556ec009d7614e273998/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/7dc869eb9e52dc675e35878465ff82e234815ec45e19700270abdf612fc2126b/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/ead4bacfeffbba11f05a3282d14069d25391d00e2aa3304877ba0be96f630822/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/7546a4be4dd82b817dfa28289f9753dbea5e37666acefd8df8cbf5931a50645a/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/3b5534059f479da2fe43ba30fd758be2682637556c0aa3efc3e33bf44ff84a5a/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/214e689cc9d7ca9025f33ff859ddce7715e8594c68ce41effe4f436d6d6ab9d4/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/15b0395f40829f4d54a6dbe007991c3436f0ec5fc8b3953422d6e92d98f368f3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/3419d9fb3de6702f64a0a4f406117fa9157f7ded0f4030b09f0227bde2179af7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/87f419ad17587c10dd1293f4f5e4363135e66fc47c42f6a8045a91bb9654364f/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/2f9536045f85192290745fe087983ab95299a9daed63ec96480630933a78c5c2/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/4a8a5f4fc1746a365b55111ce572c09d95d9e37be3599e5853dcc6e8b1151fc0/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/22eec2a011b12422ecff4596696f3d60f96ccdc710a4130bb12640b9b33164aa/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/d6ab3e07a683740c3d92c428386bde7567ba28fd80feb5c3d5f2f15c9aa27c69/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/1cd652598ab5a011d967e639605a5146a478ef3f8b75efdf3ee65dcdbb032cad/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/ea1f0b9bbad528e437e9cdcc947954a49f99de639696d9ccc259400ea14ad834/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/ca8b3b950de74f33152cfd90bd6b5453cfac9e2f719dfef7c917947c85dd1855/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/1b204a6e8c357fb12d4bc291b1a22e08d959772aaf03722aedcb50970df15644/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/f77311920a98c0b25b2476cd1d69b1c1384414cff8b70d1f2c9046b16be4cece/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/5c2420972623e181da30f613f93e61b3297cad65c6e174ed8acce244e79dea0a/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/c48041374138ca1c0076dee2decbe9825c10f157cbb78c31d11ccf359122b3e9/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/138c5e8e356b2125445bd46a43725fdb3456d7d04cfb2a373bdb86d01e27cc33/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/669aa6eda91ebeb8e7e00cd978aef0c2c4fe03f189c94b7f7182d1b7341aec98/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/126a33236acf79774584c1888e0729dd5b41fa074bf3c0e3b4931772254139f9/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/dd34c6b14d49deeb003e47d12e76a49a1606d1c87fe6eb2c739d29b2daf5f143/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/51de90229f2be610cd09ae1814bb5995f2ee10f743629f76cae19e2bc1bc0410/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/d2b9976da02f5d971ce4b1374ee54318b640c549806a86173c34d7f4ff492a41/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4424dbe356cc9ec1fa75d6de6cdd9fe45b2a1413978df2ed298b540250c79b9f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/047816e6beb161b17fd1261cb7c142d4608b04242d837f77aa03b5a224724940/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/1234b23fffa5d554551385c190395cc23ff977263bfd00fd8cba88ed19bd8baf/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/800304b45be88c2db7e5564efbbc38d4ccd6938909a9e5f3ccb5f2a60a909b41/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a6e21066a0e1da5de6eb0094c54c04b9b52227626ee986d14ab90e18bc7466bc/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b7cdfc2934c41de98e025c1940c9ab58bb8e4d86b0a15c3ab23ff3fe9d3e894c/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 03:01:48.907984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.907554 5275 manager.go:217] Machine: {Timestamp:2026-11-03 03:01:48.907091058 +0000 UTC m=+0.059779471 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100977 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea70e0b26cef491289f802562cde9 SystemUUID:ec2ea70e-0b26-cef4-9128-9f802562cde9 BootID:3b1f1784-8d01-4224-9d35-ed20e8f9ee23 Filesystems:[{Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bee6a6212f92864c57f3d0592ede4278d6b915f464afe1f6f6b1e0c00f6d17cb/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba38cb3f-dfe8-4af7-b185-943337c4759e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83cd9057-d746-4762-ba16-6bf9c83db135/volumes/kubernetes.io~projected/kube-api-access-55j87 DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:364 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b92b89af38340a65e1b92ab66c4310eb09f824adbeac71aff7f0410187b86558/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11341947e497f0d5fcfc79e6ea39ddf5015ff1a4bfbcbcbce1fc8833accdf24d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:363 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dda2da1e0579094a09522a429197c2bd081acf5ec4cbc75ba45935a42f44b31/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~projected/kube-api-access-97xxh DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53571f9a35cc96f130fa93cbb17213773b5da3d31a8a4b979acae1867f19ac8a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fbd7c0e22919c40813b5a241b357045859444d391be53f0dfc5602585072d05/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~projected/kube-api-access-6knj2 DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/865949868ecff814eebd124674043600b5d2d63ae0f45cce4598af2199e26fef/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26e509ca35bb94b0075f55604d3abf1e38455ba0334017e04f49591fbe4e2508/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62176de0bb895755cb0b185acd79a5f00e3f12d948b6c734983fa95baebae1f9/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:366 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ee1e4336-ab3b-4e70-a11c-8566fbc03510/volumes/kubernetes.io~projected/kube-api-access-xjttr DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a14dd09e-7152-49db-9326-343dc9aaaebb/volumes/kubernetes.io~projected/kube-api-access-6pbkj DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da1ca7680a1cdfb0f43754e82a228aab4a2c45163698e3b075b1e854b548506e/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~projected/kube-api-access-v8bd8 DeviceMajor:0 DeviceMinor:372 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a75ca389-0c41-4d4f-9788-da6f8bb2cc70/volumes/kubernetes.io~projected/kube-api-access-kxxs5 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~projected/kube-api-access-g4jkl DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/volumes/kubernetes.io~projected/kube-api-access-7tdrq DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6e95865-15f3-4a5d-9e0d-47a5c172ddde/volumes/kubernetes.io~projected/kube-api-access-nd9m9 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98ed7ea46690f3b13b18e90fff48e99f22a9801e279110a01c031e217e567b36/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/297990c5-51d0-4280-b974-bf6521efb03e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/549363b2eb71dc1f56fa276a3eaed4d453118862c8029932c196c0c4ee068390/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b95e773-2411-4fbc-b0f1-0e5fbde12efb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bf5ce847-feef-4cdf-b403-18eaa916c05c/volumes/kubernetes.io~projected/kube-api-access-ftswn DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fce981b0-12d5-4738-8dfe-2683c56f1a07/volumes/kubernetes.io~projected/kube-api-access-z929b DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~projected/kube-api-access-95g7q DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de6141f3-f144-46ab-86c0-d780fcb0129e/volumes/kubernetes.io~projected/kube-api-access-rffgl DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12b7e16c873c28f80781283ee52b8545f91e2f9a4af4371eaca4bbfa793a3d23/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/07201023-0dbc-4723-a3d1-d055fa9be1dc/volumes/kubernetes.io~projected/kube-api-access-nl2tn DeviceMajor:0 DeviceMinor:255 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:367 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/162d9031-a839-4af7-8ba4-973b900c618e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edc0788820823b0daba406a5b4e301eb7fc24cf84ec6719c7214cb574f419799/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4794a03d-e0ff-401e-b267-70361f49b0a0/volumes/kubernetes.io~projected/kube-api-access-258g4 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:362 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fde9d62c2ced1a044a1fa73e4dad02585c074b8239f3b7f6c83ac222f6fafa2f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53e39601-3470-4b7c-831e-5ee64bfa61ad/volumes/kubernetes.io~projected/kube-api-access-5bf2n DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/818f44c449e8499ce043b24a61d43af6f1412dd4540546fe6712c32891fdc7d5/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d221a1f-e9e2-4904-a5cb-55350a2c60bd/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf23f3-5db2-49c6-9576-a824fa35da0c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:368 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:365 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f384fb-bf15-4268-a97f-e0d537873319/volumes/kubernetes.io~projected/kube-api-access-6pvcb DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f4cf200800f5a7bf5c1ba5bf962f9d611f38c2a8b4dccb222c9831f1d0f15c8/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89fd9653f8bd6cc31d40c5428f84be52e031678a2f244a83714375dcd01b8002/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:11341947e497f0d MacAddress:be:d0:ba:2f:c4:fb Speed:10000 Mtu:8901} {Name:1dda2da1e057909 MacAddress:9a:33:ec:50:7d:0d Speed:10000 Mtu:8901} {Name:4fbd7c0e22919c4 MacAddress:1e:d1:0b:98:14:41 Speed:10000 Mtu:8901} {Name:549363b2eb71dc1 MacAddress:a6:6a:c3:67:2e:1f Speed:10000 Mtu:8901} {Name:62176de0bb89575 MacAddress:76:a6:a7:00:69:91 Speed:10000 Mtu:8901} {Name:7f4cf200800f5a7 MacAddress:8e:03:0a:05:44:10 Speed:10000 Mtu:8901} {Name:818f44c449e8499 MacAddress:7a:05:9c:29:06:50 Speed:10000 Mtu:8901} {Name:865949868ecff81 MacAddress:6a:42:f5:d7:de:c8 Speed:10000 Mtu:8901} {Name:89fd9653f8bd6cc MacAddress:26:8a:0f:54:5c:0e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:a8:91:f7:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:45:9c:c7:ad:63 Speed:0 Mtu:8901} {Name:d8cc286eaeae1f5 MacAddress:be:65:a7:8d:ac:8e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:38:a8:91:f7:bf Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:03:9f:52:26:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:39:66:95:1a:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 03:01:48.907984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.907988 5275 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 03:01:48.908265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908037 5275 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 03:01:48.908365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908341 5275 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 03:01:48.908504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908365 5275 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-7.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 03:01:48.908545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908507 5275 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 03:01:48.908545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908515 5275 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 03:01:48.908545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908526 5275 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:01:48.908619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908545 5275 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:01:48.908655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908647 5275 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 03:01:48.908822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908813 5275 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 03:01:48.908956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908947 5275 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 03:01:48.908992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.908984 5275 kubelet.go:493] "Attempting to sync node with API server" Nov 03 03:01:48.909020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.909001 5275 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 03:01:48.909020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.909012 5275 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 03:01:48.909070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.909021 5275 kubelet.go:405] "Adding apiserver pod source" Nov 03 03:01:48.909070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.909036 5275 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 03:01:48.910015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.909999 5275 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 03:01:48.910015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.910012 5275 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:01:48.910714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.910690 5275 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 03:01:48.926599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.926581 5275 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 03:01:48.926599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.926597 5275 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:01:48.926750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.926725 5275 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 03:01:48.926989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.926977 5275 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 03:01:48.927021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927009 5275 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 03:01:48.927130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927120 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 03:01:48.927160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927137 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 03:01:48.927160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927144 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 03:01:48.927160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927153 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927171 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927179 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927185 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927191 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927198 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927204 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927218 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 03:01:48.927236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927227 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 03:01:48.927461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927256 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 03:01:48.927461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.927266 5275 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 03:01:48.930811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.930794 5275 server.go:1263] "Started kubelet" Nov 03 03:01:48.930941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.930866 5275 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 03:01:48.931435 ip-10-0-136-7 systemd[1]: Started Kubernetes Kubelet. Nov 03 03:01:48.932027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.930985 5275 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 03:01:48.932027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.931520 5275 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 03:01:48.932027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.931887 5275 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 03:01:48.934548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.934524 5275 server.go:317] "Adding debug handlers to kubelet server" Nov 03 03:01:48.938535 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938515 5275 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 03:01:48.938641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938540 5275 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 03:01:48.938641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938605 5275 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-06-23 21:48:44.448811324 +0000 UTC" Nov 03 03:01:48.938641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938628 5275 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 03:01:48.938641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938638 5275 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 03:01:48.938829 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:48.938664 5275 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:01:48.938829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938732 5275 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 03:01:48.938829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.938627 5275 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14370h46m55.510189121s" Nov 03 03:01:48.939875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.939824 5275 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 03:01:48.939875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.939857 5275 factory.go:55] Registering systemd factory Nov 03 03:01:48.939875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.939866 5275 factory.go:223] Registration of the systemd container factory successfully Nov 03 03:01:48.943095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.940150 5275 factory.go:153] Registering CRI-O factory Nov 03 03:01:48.943095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.940167 5275 factory.go:223] Registration of the crio container factory successfully Nov 03 03:01:48.943095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.940187 5275 factory.go:103] Registering Raw factory Nov 03 03:01:48.943095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.940203 5275 manager.go:1199] Started watching for new ooms in manager Nov 03 03:01:48.943095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.940505 5275 manager.go:319] Starting recovery of all containers Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945288 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945333 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945355 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945370 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945384 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4794a03d-e0ff-401e-b267-70361f49b0a0" volumeName="kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945402 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162d9031-a839-4af7-8ba4-973b900c618e" volumeName="kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945417 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945436 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945454 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945473 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945485 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e95865-15f3-4a5d-9e0d-47a5c172ddde" volumeName="kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945497 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e95865-15f3-4a5d-9e0d-47a5c172ddde" volumeName="kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945513 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945528 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945548 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945557 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53e39601-3470-4b7c-831e-5ee64bfa61ad" volumeName="kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945576 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945585 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945596 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945613 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297990c5-51d0-4280-b974-bf6521efb03e" volumeName="kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945622 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945634 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945643 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945652 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945664 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" volumeName="kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945673 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945691 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba38cb3f-dfe8-4af7-b185-943337c4759e" volumeName="kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945700 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945725 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945738 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4794a03d-e0ff-401e-b267-70361f49b0a0" volumeName="kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945758 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83cd9057-d746-4762-ba16-6bf9c83db135" volumeName="kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945776 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" volumeName="kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945790 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee1e4336-ab3b-4e70-a11c-8566fbc03510" volumeName="kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945807 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945822 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162d9031-a839-4af7-8ba4-973b900c618e" volumeName="kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945839 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945891 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4794a03d-e0ff-401e-b267-70361f49b0a0" volumeName="kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.945908 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad" volumeName="kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946020 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf5ce847-feef-4cdf-b403-18eaa916c05c" volumeName="kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946034 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946084 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf5ce847-feef-4cdf-b403-18eaa916c05c" volumeName="kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946098 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946155 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14dd09e-7152-49db-9326-343dc9aaaebb" volumeName="kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946168 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce981b0-12d5-4738-8dfe-2683c56f1a07" volumeName="kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946198 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946228 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14dd09e-7152-49db-9326-343dc9aaaebb" volumeName="kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946240 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946257 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946301 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4794a03d-e0ff-401e-b267-70361f49b0a0" volumeName="kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946314 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d221a1f-e9e2-4904-a5cb-55350a2c60bd" volumeName="kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946331 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad" volumeName="kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946372 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" volumeName="kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946405 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946418 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" volumeName="kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946444 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6141f3-f144-46ab-86c0-d780fcb0129e" volumeName="kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946462 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946494 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce981b0-12d5-4738-8dfe-2683c56f1a07" volumeName="kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946520 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b95e773-2411-4fbc-b0f1-0e5fbde12efb" volumeName="kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946536 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946548 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" volumeName="kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946597 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" volumeName="kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946609 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946621 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946661 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946917 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 03:01:48.947014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.946941 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947064 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e95865-15f3-4a5d-9e0d-47a5c172ddde" volumeName="kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947108 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f384fb-bf15-4268-a97f-e0d537873319" volumeName="kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947125 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162d9031-a839-4af7-8ba4-973b900c618e" volumeName="kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947145 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b95e773-2411-4fbc-b0f1-0e5fbde12efb" volumeName="kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947157 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53e39601-3470-4b7c-831e-5ee64bfa61ad" volumeName="kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947175 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14dd09e-7152-49db-9326-343dc9aaaebb" volumeName="kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947189 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14dd09e-7152-49db-9326-343dc9aaaebb" volumeName="kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947202 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947221 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07201023-0dbc-4723-a3d1-d055fa9be1dc" volumeName="kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947234 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53e39601-3470-4b7c-831e-5ee64bfa61ad" volumeName="kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947251 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf23f3-5db2-49c6-9576-a824fa35da0c" volumeName="kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947262 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297990c5-51d0-4280-b974-bf6521efb03e" volumeName="kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947274 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f110ed-1cbf-478a-a45f-c2181a1a5563" volumeName="kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947292 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53e39601-3470-4b7c-831e-5ee64bfa61ad" volumeName="kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947303 5275 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6141f3-f144-46ab-86c0-d780fcb0129e" volumeName="kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl" seLinuxMountContext="" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947314 5275 reconstruct.go:97] "Volume reconstruction finished" Nov 03 03:01:48.949768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.947321 5275 reconciler.go:29] "Reconciler: start to sync state" Nov 03 03:01:48.968353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.968325 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:48.968697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.968677 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:48.975249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.975179 5275 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 03:01:48.976507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.976491 5275 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 03:01:48.976507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.976507 5275 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 03:01:48.976605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.976523 5275 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 03:01:48.976605 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:48.976556 5275 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 03:01:48.995987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:48.995952 5275 generic.go:356] "Generic (PLEG): container finished" podID="d50507705f9f63f8e4597fb6868abd9a" containerID="11bcdfef753f14737486560a132ccd4c58a96e4b62574fab608e824938d61060" exitCode=0 Nov 03 03:01:49.018506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018466 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="c8fd460ffd92bdc0566b3c0382e821051e87799692dcd2af7578151ecdbb36ef" exitCode=0 Nov 03 03:01:49.018506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018503 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="2f7b421751416be35d03fc82946970875b1b409bfab6886477bf4bb50a32b063" exitCode=0 Nov 03 03:01:49.018670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018516 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="7616291c30875b2efbe3abde2f907166ececc0cb531bd068dc3079c0e436a096" exitCode=0 Nov 03 03:01:49.018670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018526 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="e10aaf933a32822625405fae1f94c75c79ab35f29869f2c122f53f39a7f94933" exitCode=0 Nov 03 03:01:49.018670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018534 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="d687f62c7ea7674a7439203ae6c8f375f4cf32e425e10b999046761a1a375fe4" exitCode=0 Nov 03 03:01:49.018670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.018545 5275 generic.go:356] "Generic (PLEG): container finished" podID="4794a03d-e0ff-401e-b267-70361f49b0a0" containerID="361c6ad97554fbf33f8476e06b9c2549c0dcecf22a1926c5b417bed3b4b10686" exitCode=0 Nov 03 03:01:49.023005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.022986 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:01:49.023353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.023324 5275 generic.go:356] "Generic (PLEG): container finished" podID="a75ca389-0c41-4d4f-9788-da6f8bb2cc70" containerID="225c6d7deef9a8da3b05acfe8f542a658b6eac56e4fd9838b4dd38499eb22b72" exitCode=1 Nov 03 03:01:49.028570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.028547 5275 generic.go:356] "Generic (PLEG): container finished" podID="f0f384fb-bf15-4268-a97f-e0d537873319" containerID="efea0e4285dd0bd2b6a631e6bd4b1f1fd024e3c49aa2fefc2a020acd24c99233" exitCode=0 Nov 03 03:01:49.038726 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.038706 5275 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:01:49.040567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.040519 5275 manager.go:324] Recovery completed Nov 03 03:01:49.060528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.060506 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.062524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.062508 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.062589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.062539 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.062589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.062549 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.067889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.067869 5275 cpu_manager.go:226] "Starting" policy="none" Nov 03 03:01:49.067889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.067883 5275 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 03:01:49.068041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.067884 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.068041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.067903 5275 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068039 5275 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068050 5275 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068080 5275 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068085 5275 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068091 5275 policy_none.go:50] "Start" Nov 03 03:01:49.068111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068100 5275 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 03:01:49.068270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068112 5275 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 03:01:49.068270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068249 5275 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 03:01:49.068270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068256 5275 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 03:01:49.068270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068262 5275 policy_none.go:44] "Start" Nov 03 03:01:49.068949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068928 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.069073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068963 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.069073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.068979 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.073210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.073179 5275 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 03:01:49.075978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.075928 5275 manager.go:341] "Starting Device Plugin manager" Nov 03 03:01:49.075978 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.075954 5275 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 03:01:49.076120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076056 5275 server.go:87] "Starting device plugin registration server" Nov 03 03:01:49.076511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076421 5275 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 03:01:49.076511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076437 5275 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 03:01:49.076649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076611 5275 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal"] Nov 03 03:01:49.076739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076723 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.076832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076809 5275 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 03:01:49.076989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.076950 5275 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 03:01:49.077068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.077000 5275 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 03:01:49.077068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.077008 5275 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 03:01:49.077163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.077062 5275 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:01:49.076986866Z","UUID":"279791cc-8756-4c34-97be-725882522a4c","Handler":null,"Name":"","Endpoint":""} Nov 03 03:01:49.079432 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.079381 5275 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 03:01:49.080733 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.080711 5275 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" Nov 03 03:01:49.080733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.080717 5275 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.080766 5275 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.080769 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.080798 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.080821 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.081451 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.082126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.081788 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.082750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082731 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.082919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082785 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.082919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082802 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.082919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082835 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.082919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082882 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.082919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.082911 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.083154 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.083140 5275 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.083459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083424 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerStarted","Data":"ee57a2b65703cb1be46b8147d6fc24cfb9da74f8c890ef65108d213a58b5d04d"} Nov 03 03:01:49.083518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083468 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerDied","Data":"11bcdfef753f14737486560a132ccd4c58a96e4b62574fab608e824938d61060"} Nov 03 03:01:49.083518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083485 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"d50507705f9f63f8e4597fb6868abd9a","Type":"ContainerStarted","Data":"1d81ce0b914f21509c3ee41670c6bd1026732721d9e602f5e7a45d0630502e3a"} Nov 03 03:01:49.083609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083587 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"b7e862e4bcfcee2fc388ea5939653511","Type":"ContainerStarted","Data":"7cb35b4a9db4e6b2b2db6a2ab570cf2b8de78b5780beba3dafc1dd957a588240"} Nov 03 03:01:49.083644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083626 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" event={"ID":"b7e862e4bcfcee2fc388ea5939653511","Type":"ContainerStarted","Data":"fc37b6953b563c70aeaddab8dfec997625dc315e6b29403ddf7325d8da9835fa"} Nov 03 03:01:49.083690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.083599 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.085909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.085889 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.086010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.085911 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.086010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.085925 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.086102 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.086051 5275 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-7.us-west-1.compute.internal\" not found" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.122024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.121985 5275 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:49.122024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.121985 5275 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:49.122250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.122103 5275 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 03:01:49.174256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.174225 5275 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:49.177208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.177191 5275 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:49.179265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.179245 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:49.179347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.179277 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:49.179347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.179288 5275 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-7.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:49.179347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.179330 5275 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.219124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.219093 5275 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:49.222364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.222340 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.222477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.222370 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.222477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.222392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.238935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.238907 5275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.292058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.291985 5275 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.292058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.292060 5275 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.303910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.303882 5275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:49.304040 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.303932 5275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.304040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.303946 5275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322516 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322543 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322568 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322617 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322626 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d50507705f9f63f8e4597fb6868abd9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"d50507705f9f63f8e4597fb6868abd9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.322728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.322666 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e862e4bcfcee2fc388ea5939653511-config\") pod \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" (UID: \"b7e862e4bcfcee2fc388ea5939653511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.384116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.384085 5275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:49.384266 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:49.384135 5275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:49.909347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:49.909295 5275 apiserver.go:52] "Watching apiserver" Nov 03 03:01:50.036037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.036015 5275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:50.036181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.036155 5275 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:50.080731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.080700 5275 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 03:01:50.081369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.081341 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb","openshift-dns/dns-default-xv2gh","openshift-monitoring/thanos-querier-6cb88f77ff-knhmq","openshift-network-diagnostics/network-check-target-jk4nq","openshift-monitoring/node-exporter-mgf2f","openshift-ovn-kubernetes/ovnkube-node-gvtt8","openshift-cluster-node-tuning-operator/tuned-vg788","openshift-dns/node-resolver-ncxp7","kube-system/konnectivity-agent-v499x","openshift-network-operator/iptables-alerter-4gfm4","openshift-ingress-canary/ingress-canary-thwdd","openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b","openshift-monitoring/prometheus-k8s-0","openshift-insights/insights-runtime-extractor-mtw4b","openshift-image-registry/node-ca-tdbsm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal","openshift-multus/multus-tgpmj","kube-system/global-pull-secret-syncer-4vzrm","openshift-multus/network-metrics-daemon-4c62k","openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75","openshift-multus/multus-additional-cni-plugins-qwcqm"] Nov 03 03:01:50.114472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.114439 5275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:50.116118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.116103 5275 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:50.128744 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:50.128713 5275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:50.128744 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:01:50.128737 5275 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal" Nov 03 03:01:50.141659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.141636 5275 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 03:01:50.166885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.166791 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 03:01:50.169483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169419 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 03:01:50.169483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169428 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\"" Nov 03 03:01:50.169667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169573 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\"" Nov 03 03:01:50.169787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169740 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\"" Nov 03 03:01:50.169945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169928 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 03:01:50.169945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.169929 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\"" Nov 03 03:01:50.170525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.170505 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 03:01:50.170698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.170683 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\"" Nov 03 03:01:50.170872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.170840 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\"" Nov 03 03:01:50.171754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.171724 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 03:01:50.173484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173458 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\"" Nov 03 03:01:50.173595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173488 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\"" Nov 03 03:01:50.173595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173503 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 03:01:50.173595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173533 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173610 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173612 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173635 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173648 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173671 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173688 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\"" Nov 03 03:01:50.173761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.173707 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\"" Nov 03 03:01:50.179351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179336 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 03:01:50.179568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179551 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.179568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179564 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 03:01:50.179779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179763 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.179863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179796 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.179900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.179892 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 03:01:50.180021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180007 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 03:01:50.180072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180057 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 03:01:50.180116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180078 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.180207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180190 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 03:01:50.180255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180195 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.180255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180213 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.180255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180235 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 03:01:50.180408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.180394 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.229016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.228982 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.229016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229017 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.229223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229038 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.229223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229057 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.229223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229122 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdrq\" (UniqueName: \"kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:50.229223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229217 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.229434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229292 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.229434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229346 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.229434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229413 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.229547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229465 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-tmp\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.229547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229481 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.229547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229533 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.229649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229572 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.229649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229615 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce981b0-12d5-4738-8dfe-2683c56f1a07-tmp-dir\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.229649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229622 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.229746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229684 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de6141f3-f144-46ab-86c0-d780fcb0129e-iptables-alerter-script\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.229786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229747 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.229786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229767 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.229879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229787 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftswn\" (UniqueName: \"kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.229879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229801 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cni-binary-copy\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.229879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229811 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.229879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229869 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.230009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.229961 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.230053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230005 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z929b\" (UniqueName: \"kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.230053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230042 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.230127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230067 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.230127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230115 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.230182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230146 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.230182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230163 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.230264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230209 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.230329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230311 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.230387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230340 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.230387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230361 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95g7q\" (UniqueName: \"kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:50.230387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230383 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.230492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230398 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.230492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230412 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.230492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230473 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.230591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230499 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.230591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230548 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.230591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230577 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.230692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230580 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a14dd09e-7152-49db-9326-343dc9aaaebb-data-volume\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.230692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230595 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffgl\" (UniqueName: \"kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.230692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230629 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.230787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230749 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knj2\" (UniqueName: \"kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.230787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230772 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvcb\" (UniqueName: \"kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.230886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230821 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.230931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.230931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230905 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf5ce847-feef-4cdf-b403-18eaa916c05c-serviceca\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.230931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230925 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.231055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230959 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.231055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.230995 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231027 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231068 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231106 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231133 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.231190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231160 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.231250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231192 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.231250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231207 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231224 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.231358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231252 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.231358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231283 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.231358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231313 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231351 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258g4\" (UniqueName: \"kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.231502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231385 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.231654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231616 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.231777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231658 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.231777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231695 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.231777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231741 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.231957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231782 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:50.231957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231890 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.231957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231918 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.231957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231945 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231970 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.231993 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-metrics-certs\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232037 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/07201023-0dbc-4723-a3d1-d055fa9be1dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232041 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232075 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232117 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.232142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232141 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232147 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53e39601-3470-4b7c-831e-5ee64bfa61ad-metrics-tls\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232183 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232224 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232264 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232285 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232335 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232358 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9m9\" (UniqueName: \"kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.232417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232412 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232444 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbkj\" (UniqueName: \"kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232470 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232493 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232516 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232549 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232575 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232587 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e39601-3470-4b7c-831e-5ee64bfa61ad-tmp-dir\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232631 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxs5\" (UniqueName: \"kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232661 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232677 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.232745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232729 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232767 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232818 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-textfile\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232914 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232952 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.232977 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233001 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233034 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233079 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233103 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cf23f3-5db2-49c6-9576-a824fa35da0c-tmp\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233133 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233200 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.233246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233233 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233255 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233308 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjttr\" (UniqueName: \"kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233340 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233378 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233395 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233428 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233578 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297990c5-51d0-4280-b974-bf6521efb03e-cert\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233585 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233618 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233648 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-konnectivity-ca\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233652 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovn-node-metrics-cert\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233650 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233704 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.233742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233734 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233760 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233789 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2tn\" (UniqueName: \"kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233813 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233872 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-config\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233949 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233978 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.233978 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a14dd09e-7152-49db-9326-343dc9aaaebb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234037 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234086 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234121 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234146 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234172 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234198 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234263 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba38cb3f-dfe8-4af7-b185-943337c4759e-original-pull-secret\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/162d9031-a839-4af7-8ba4-973b900c618e-etc-tuned\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234291 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf2n\" (UniqueName: \"kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234304 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234323 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jkl\" (UniqueName: \"kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234343 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234365 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.234401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234392 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234418 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234436 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234445 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234471 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234489 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234556 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xxh\" (UniqueName: \"kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234573 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234590 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234605 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234611 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234644 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234661 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234785 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4794a03d-e0ff-401e-b267-70361f49b0a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.235164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.234825 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b95e773-2411-4fbc-b0f1-0e5fbde12efb-agent-certs\") pod \"konnectivity-agent-v499x\" (UID: \"4b95e773-2411-4fbc-b0f1-0e5fbde12efb\") " pod="kube-system/konnectivity-agent-v499x" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257395 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257468 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257479 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257513 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257660 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.257749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257741 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.258169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.257954 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.260858 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.260823 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 03:01:50.261373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261355 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.261453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261430 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.261501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261447 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 03:01:50.261501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261457 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.261501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261454 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261502 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261519 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261524 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261548 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261552 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261596 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 03:01:50.261633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261597 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 03:01:50.261946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261667 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 03:01:50.261946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.261740 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 03:01:50.262051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262034 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 03:01:50.262093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262035 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 03:01:50.262181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262164 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-env-overrides\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.262231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262186 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.262231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262213 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.262457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262436 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 03:01:50.262517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.262445 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 03:01:50.263159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.263140 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-ovnkube-script-lib\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.264088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.264072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.264728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.264710 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.264770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.264713 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.264872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.264838 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-daemon-config\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.269962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.269946 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.271060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.271046 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53e39601-3470-4b7c-831e-5ee64bfa61ad-config-volume\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.271128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.271081 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.271128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.271090 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92cf23f3-5db2-49c6-9576-a824fa35da0c-metrics-client-ca\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.272217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.272202 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-metrics-client-ca\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.272262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.272213 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.305486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.305449 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 03:01:50.305651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.305554 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 03:01:50.305651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.305599 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 03:01:50.306769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.306748 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 03:01:50.306939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.306754 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 03:01:50.306939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.306873 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 03:01:50.307064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307037 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 03:01:50.307296 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307277 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gdrqn\"" Nov 03 03:01:50.307376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307286 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 03:01:50.307376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307313 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 03:01:50.307376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307335 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 03:01:50.307376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307356 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-48h7s1kcmmk62\"" Nov 03 03:01:50.307376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307365 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307381 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307392 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-crpwp\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307416 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307382 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307512 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307523 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307530 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307562 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ccc7x\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307569 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307574 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dh9lv\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307577 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cn4p\"" Nov 03 03:01:50.307684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.307590 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 03:01:50.309737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.309721 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.310596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310577 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.310693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310675 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.310802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310782 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.310880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310786 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-grpc-tls\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.310975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310876 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.310975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310894 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 03:01:50.310975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310818 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 03:01:50.310975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.310950 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ba8kqj028vm5e\"" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311377 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvcb\" (UniqueName: \"kubernetes.io/projected/f0f384fb-bf15-4268-a97f-e0d537873319-kube-api-access-6pvcb\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf2n\" (UniqueName: \"kubernetes.io/projected/53e39601-3470-4b7c-831e-5ee64bfa61ad-kube-api-access-5bf2n\") pod \"dns-default-xv2gh\" (UID: \"53e39601-3470-4b7c-831e-5ee64bfa61ad\") " pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311458 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311527 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjttr\" (UniqueName: \"kubernetes.io/projected/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kube-api-access-xjttr\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311531 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311557 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311529 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258g4\" (UniqueName: \"kubernetes.io/projected/4794a03d-e0ff-401e-b267-70361f49b0a0-kube-api-access-258g4\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311662 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.311400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311687 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.312068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311750 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.312068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311818 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95g7q\" (UniqueName: \"kubernetes.io/projected/7d221a1f-e9e2-4904-a5cb-55350a2c60bd-kube-api-access-95g7q\") pod \"network-metrics-daemon-4c62k\" (UID: \"7d221a1f-e9e2-4904-a5cb-55350a2c60bd\") " pod="openshift-multus/network-metrics-daemon-4c62k" Nov 03 03:01:50.312068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311818 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.312068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.311994 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xxh\" (UniqueName: \"kubernetes.io/projected/162d9031-a839-4af7-8ba4-973b900c618e-kube-api-access-97xxh\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.312251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.312233 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j87\" (UniqueName: \"kubernetes.io/projected/83cd9057-d746-4762-ba16-6bf9c83db135-kube-api-access-55j87\") pod \"network-check-target-jk4nq\" (UID: \"83cd9057-d746-4762-ba16-6bf9c83db135\") " pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:01:50.312701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.312686 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.312894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.312878 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.313042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.313024 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.313413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.313356 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.313494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.313476 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2tn\" (UniqueName: \"kubernetes.io/projected/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-api-access-nl2tn\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.313552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.313509 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.313773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.313757 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.314091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314073 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z929b\" (UniqueName: \"kubernetes.io/projected/fce981b0-12d5-4738-8dfe-2683c56f1a07-kube-api-access-z929b\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.314578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314559 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07201023-0dbc-4723-a3d1-d055fa9be1dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bdb75\" (UID: \"07201023-0dbc-4723-a3d1-d055fa9be1dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bdb75" Nov 03 03:01:50.314739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314600 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:50.314739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314705 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-tls\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.314823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314741 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92cf23f3-5db2-49c6-9576-a824fa35da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.314877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.314836 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.316515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.316497 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.318497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.318477 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9m9\" (UniqueName: \"kubernetes.io/projected/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-kube-api-access-nd9m9\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.318570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.318530 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxs5\" (UniqueName: \"kubernetes.io/projected/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-kube-api-access-kxxs5\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.318690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.318670 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.318782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.318767 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbkj\" (UniqueName: \"kubernetes.io/projected/a14dd09e-7152-49db-9326-343dc9aaaebb-kube-api-access-6pbkj\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.318908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.318893 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knj2\" (UniqueName: \"kubernetes.io/projected/92cf23f3-5db2-49c6-9576-a824fa35da0c-kube-api-access-6knj2\") pod \"thanos-querier-6cb88f77ff-knhmq\" (UID: \"92cf23f3-5db2-49c6-9576-a824fa35da0c\") " pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:50.319762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.319746 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffgl\" (UniqueName: \"kubernetes.io/projected/de6141f3-f144-46ab-86c0-d780fcb0129e-kube-api-access-rffgl\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.320662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.320646 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jkl\" (UniqueName: \"kubernetes.io/projected/297990c5-51d0-4280-b974-bf6521efb03e-kube-api-access-g4jkl\") pod \"ingress-canary-thwdd\" (UID: \"297990c5-51d0-4280-b974-bf6521efb03e\") " pod="openshift-ingress-canary/ingress-canary-thwdd" Nov 03 03:01:50.320792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.320776 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:50.323802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.323769 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftswn\" (UniqueName: \"kubernetes.io/projected/bf5ce847-feef-4cdf-b403-18eaa916c05c-kube-api-access-ftswn\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.329858 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.329823 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdrq\" (UniqueName: \"kubernetes.io/projected/81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad-kube-api-access-7tdrq\") pod \"monitoring-plugin-77f4f76594-2wc4b\" (UID: \"81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:50.335744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335723 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335757 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335782 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335815 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335839 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335839 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-node-log\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.335876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335870 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335886 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335908 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335922 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-ovn\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335927 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-etc-selinux\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335974 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-netns\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335981 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335985 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-sys\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335932 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a14dd09e-7152-49db-9326-343dc9aaaebb-crio-socket\") pod \"insights-runtime-extractor-mtw4b\" (UID: \"a14dd09e-7152-49db-9326-343dc9aaaebb\") " pod="openshift-insights/insights-runtime-extractor-mtw4b" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336023 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.335939 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-netd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336046 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336087 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-node-exporter-wtmp\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336093 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysconfig\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336083 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336145 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336164 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336174 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-run\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336198 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336205 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-log-socket\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336216 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336263 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336285 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-multus\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336299 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336315 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336330 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336341 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-registration-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336348 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-kubelet\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336350 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336362 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-socket-dir-parent\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336388 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-sys-fs\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336387 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336396 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-system-cni-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336403 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de6141f3-f144-46ab-86c0-d780fcb0129e-host-slash\") pod \"iptables-alerter-4gfm4\" (UID: \"de6141f3-f144-46ab-86c0-d780fcb0129e\") " pod="openshift-network-operator/iptables-alerter-4gfm4" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336417 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336421 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-modprobe-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336421 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336448 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-cni-bin\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336460 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336464 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-conf\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336476 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336491 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336500 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-dbus\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336507 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336530 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336557 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-systemd-units\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336568 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-host\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336576 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fce981b0-12d5-4738-8dfe-2683c56f1a07-hosts-file\") pod \"node-resolver-ncxp7\" (UID: \"fce981b0-12d5-4738-8dfe-2683c56f1a07\") " pod="openshift-dns/node-resolver-ncxp7" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336606 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336622 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-lib-modules\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336639 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336662 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336697 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-sysctl-d\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336696 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336718 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf5ce847-feef-4cdf-b403-18eaa916c05c-host\") pod \"node-ca-tdbsm\" (UID: \"bf5ce847-feef-4cdf-b403-18eaa916c05c\") " pod="openshift-image-registry/node-ca-tdbsm" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336724 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.336723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336750 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336772 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-cnibin\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336775 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336792 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-sys\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336795 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-multus-certs\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336774 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-hostroot\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336817 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-var-lib-kubelet\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336832 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336879 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0f384fb-bf15-4268-a97f-e0d537873319-root\") pod \"node-exporter-mgf2f\" (UID: \"f0f384fb-bf15-4268-a97f-e0d537873319\") " pod="openshift-monitoring/node-exporter-mgf2f" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336888 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336909 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336924 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336939 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336955 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336975 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-device-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336979 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-kubernetes\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336979 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.336993 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-etc-kubernetes\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337010 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337018 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/162d9031-a839-4af7-8ba4-973b900c618e-etc-systemd\") pod \"tuned-vg788\" (UID: \"162d9031-a839-4af7-8ba4-973b900c618e\") " pod="openshift-cluster-node-tuning-operator/tuned-vg788" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337026 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337029 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337045 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337048 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-kubelet\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337027 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-host-slash\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337074 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337091 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337104 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337123 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-systemd\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337124 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337142 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-var-lib-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337159 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337169 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-netns\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337183 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337201 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337261 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-etc-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337308 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337334 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337339 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-cnibin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337369 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4794a03d-e0ff-401e-b267-70361f49b0a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qwcqm\" (UID: \"4794a03d-e0ff-401e-b267-70361f49b0a0\") " pod="openshift-multus/multus-additional-cni-plugins-qwcqm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337369 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337379 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba38cb3f-dfe8-4af7-b185-943337c4759e-kubelet-config\") pod \"global-pull-secret-syncer-4vzrm\" (UID: \"ba38cb3f-dfe8-4af7-b185-943337c4759e\") " pod="kube-system/global-pull-secret-syncer-4vzrm" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337392 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337401 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a75ca389-0c41-4d4f-9788-da6f8bb2cc70-run-openvswitch\") pod \"ovnkube-node-gvtt8\" (UID: \"a75ca389-0c41-4d4f-9788-da6f8bb2cc70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337421 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337441 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337446 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-run-k8s-cni-cncf-io\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337455 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337482 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-multus-conf-dir\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337505 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee1e4336-ab3b-4e70-a11c-8566fbc03510-socket-dir\") pod \"aws-ebs-csi-driver-node-c66sb\" (UID: \"ee1e4336-ab3b-4e70-a11c-8566fbc03510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c66sb" Nov 03 03:01:50.338095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.337510 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e95865-15f3-4a5d-9e0d-47a5c172ddde-host-var-lib-cni-bin\") pod \"multus-tgpmj\" (UID: \"d6e95865-15f3-4a5d-9e0d-47a5c172ddde\") " pod="openshift-multus/multus-tgpmj" Nov 03 03:01:50.395489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.395471 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:01:50.599090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.599059 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:50.602700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:50.602667 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2wc4b" Nov 03 03:01:52.004911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:52.004876 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xv2gh" Nov 03 03:01:53.048199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:53.048166 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" exitCode=0 Nov 03 03:01:53.048630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:53.048252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} Nov 03 03:01:53.512972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:53.512898 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:54.023816 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:54.023788 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:01:54.052306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:54.052278 5275 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 03:01:54.378002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:54.377914 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:54.384402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:54.384375 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cb88f77ff-knhmq" Nov 03 03:01:59.073792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073752 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} Nov 03 03:01:59.074212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073829 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} Nov 03 03:01:59.074212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073877 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} Nov 03 03:01:59.074212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073912 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} Nov 03 03:01:59.074212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073953 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} Nov 03 03:01:59.074212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.073989 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerStarted","Data":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} Nov 03 03:01:59.710129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.710090 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:01:59.943053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:01:59.943021 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:02:02.328119 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:02:02.328083 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:02:03.679738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:03.679699 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvtt8" Nov 03 03:02:12.569115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:12.569081 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v499x" Nov 03 03:02:19.946445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:19.946412 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jk4nq" Nov 03 03:02:32.327494 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:02:32.327451 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:02:37.459977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:37.459945 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:02:37.480084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:37.480057 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:02:38.222109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:38.222081 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:02:59.879916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.879879 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:02:59.880928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.880886 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="prometheus" containerID="cri-o://9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" gracePeriod=600 Nov 03 03:02:59.880993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.880936 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy" containerID="cri-o://f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" gracePeriod=600 Nov 03 03:02:59.880993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.880962 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="config-reloader" containerID="cri-o://5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" gracePeriod=600 Nov 03 03:02:59.881067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.880890 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-web" containerID="cri-o://961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" gracePeriod=600 Nov 03 03:02:59.881067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.880889 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-thanos" containerID="cri-o://48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" gracePeriod=600 Nov 03 03:02:59.881154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:02:59.881064 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="thanos-sidecar" containerID="cri-o://38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" gracePeriod=600 Nov 03 03:03:00.127377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.127352 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.237546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237450 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237498 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237528 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237558 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237684 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237731 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237765 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237794 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.237831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237819 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237880 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237918 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237947 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237949 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.237980 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238037 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238066 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238093 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238095 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238105 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.238159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238121 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.238640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238617 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.238951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238919 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.239050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.238984 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.239574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.239276 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:03:00.240692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.240663 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.240790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.240734 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.240790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.240768 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") pod \"29f110ed-1cbf-478a-a45f-c2181a1a5563\" (UID: \"29f110ed-1cbf-478a-a45f-c2181a1a5563\") " Nov 03 03:03:00.241027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241009 5275 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-metrics-client-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241034 5275 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241048 5275 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241063 5275 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241075 5275 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241087 5275 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-db\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241099 5275 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.241625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.241145 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.242409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.242381 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.242558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.242528 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.242728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.242697 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.242900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.242876 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "kube-api-access-v8bd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:03:00.242991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.242935 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:03:00.243186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243163 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:03:00.243316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243300 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.243452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243427 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:03:00.243571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243550 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.243616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243602 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.243830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.243813 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.252136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.252111 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config" pod "29f110ed-1cbf-478a-a45f-c2181a1a5563" (UID: "29f110ed-1cbf-478a-a45f-c2181a1a5563"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:03:00.280226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280190 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" exitCode=0 Nov 03 03:03:00.280226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280221 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" exitCode=0 Nov 03 03:03:00.280226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280230 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" exitCode=0 Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280239 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" exitCode=0 Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280248 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" exitCode=0 Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280256 5275 generic.go:356] "Generic (PLEG): container finished" podID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" exitCode=0 Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280287 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280285 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280356 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280382 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280409 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280430 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280434 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280458 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} Nov 03 03:03:00.280507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.280480 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29f110ed-1cbf-478a-a45f-c2181a1a5563","Type":"ContainerDied","Data":"d8cc286eaeae1f5bfbcf8e0cf2632feee8a90412fe8083b7e5463c0c0c03c49a"} Nov 03 03:03:00.288978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.288959 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.295793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.295771 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.302505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.302486 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.309916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.309900 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.316972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.316949 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.325342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.325323 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.332213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.332196 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.332502 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.332480 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.332590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.332511 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.332590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.332546 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.332869 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.332820 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.332942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.332868 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.332942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.332885 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.333129 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.333112 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.333214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333134 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.333214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333149 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.333401 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.333381 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.333463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333405 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.333463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333423 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.333669 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.333650 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.333731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333672 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.333731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333688 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.333978 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.333959 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.334047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333980 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.334047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.333993 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.334242 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:00.334224 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.334306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334247 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.334306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334265 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.334497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334475 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.334551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334499 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.334709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334693 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.334760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334709 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.334921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334905 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.334921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.334920 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.335185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335148 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.335185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335172 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.335460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335434 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.335460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335456 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.335723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335699 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.335723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335722 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.335987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335971 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.335987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.335987 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.336277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336257 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.336277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336272 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.336518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336502 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.336518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336518 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.336760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336742 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.336760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.336760 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.337044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337025 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.337113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337045 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.337305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337287 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.337369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337305 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.337574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337557 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.337617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337575 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.337807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337790 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.337866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.337808 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.338061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338045 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.338101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338062 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.338263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338246 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.338298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338264 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.338496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338480 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.338544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338497 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.338715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338698 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.338759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338716 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.338957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338937 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.339006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.338959 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.339186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339169 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.339233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339186 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.339406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339391 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.339453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339407 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.339579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339564 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.339579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339578 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.339768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339751 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.339808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339769 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.340015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.339998 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.340067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340015 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.340211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340196 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.340248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340210 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.340431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340414 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.340473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340432 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.340630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340614 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.340678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340631 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.340858 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340828 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.340911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.340859 5275 scope.go:122] "RemoveContainer" containerID="48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa" Nov 03 03:03:00.341047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341032 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa"} err="failed to get container status \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": rpc error: code = NotFound desc = could not find container \"48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa\": container with ID starting with 48cc02d254c997d87e790e2ea0b470daefd2c64cfe7834fa3337fc753d1605fa not found: ID does not exist" Nov 03 03:03:00.341099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341048 5275 scope.go:122] "RemoveContainer" containerID="f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f" Nov 03 03:03:00.341229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341211 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f"} err="failed to get container status \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": rpc error: code = NotFound desc = could not find container \"f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f\": container with ID starting with f356f681fddf81d4d3dcf8316fe62f2270ddd1ad734085b796ea6dbfcdf2bf6f not found: ID does not exist" Nov 03 03:03:00.341275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341231 5275 scope.go:122] "RemoveContainer" containerID="961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3" Nov 03 03:03:00.341421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341402 5275 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-kube-rbac-proxy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341426 5275 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-tls-assets\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341441 5275 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-grpc-tls\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341432 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3"} err="failed to get container status \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": rpc error: code = NotFound desc = could not find container \"961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3\": container with ID starting with 961855cb2760c06e9d1a39dcf71a111213dbcef7333208f94644f499b03470c3 not found: ID does not exist" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341455 5275 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341466 5275 scope.go:122] "RemoveContainer" containerID="38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341471 5275 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29f110ed-1cbf-478a-a45f-c2181a1a5563-config-out\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341485 5275 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341499 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bd8\" (UniqueName: \"kubernetes.io/projected/29f110ed-1cbf-478a-a45f-c2181a1a5563-kube-api-access-v8bd8\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341514 5275 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-web-config\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341529 5275 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-metrics-client-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341542 5275 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-config\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341553 5275 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29f110ed-1cbf-478a-a45f-c2181a1a5563-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341563 5275 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341573 5275 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29f110ed-1cbf-478a-a45f-c2181a1a5563-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341662 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8"} err="failed to get container status \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": rpc error: code = NotFound desc = could not find container \"38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8\": container with ID starting with 38d12dcc335e1e0e387999b7b3eef699e3b49f4109c8bee60f4e519e474ff8f8 not found: ID does not exist" Nov 03 03:03:00.341769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341676 5275 scope.go:122] "RemoveContainer" containerID="5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8" Nov 03 03:03:00.342060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341836 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8"} err="failed to get container status \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": rpc error: code = NotFound desc = could not find container \"5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8\": container with ID starting with 5d7428a55e57c07ab36a6015618ac16baa74c3734a02e7cf49a093f9545f95d8 not found: ID does not exist" Nov 03 03:03:00.342060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.341867 5275 scope.go:122] "RemoveContainer" containerID="9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13" Nov 03 03:03:00.342129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.342059 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13"} err="failed to get container status \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": rpc error: code = NotFound desc = could not find container \"9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13\": container with ID starting with 9bb6792a0fff4a1082d6bc487117c708b2edab972088d128e7b076c4b5ee9a13 not found: ID does not exist" Nov 03 03:03:00.342129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.342071 5275 scope.go:122] "RemoveContainer" containerID="14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e" Nov 03 03:03:00.342255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.342239 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e"} err="failed to get container status \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": rpc error: code = NotFound desc = could not find container \"14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e\": container with ID starting with 14d745b01c9ffa700a0c075c11fd1c39c38f4cebf1a6970019f5f9afff2cc65e not found: ID does not exist" Nov 03 03:03:00.450586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.450554 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:03:00.455036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.455006 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:03:00.488619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488542 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:03:00.488819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488806 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="prometheus" Nov 03 03:03:00.488884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488820 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="prometheus" Nov 03 03:03:00.488884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488836 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-web" Nov 03 03:03:00.488884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488842 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-web" Nov 03 03:03:00.488884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488863 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="init-config-reloader" Nov 03 03:03:00.488884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488872 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="init-config-reloader" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488887 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-thanos" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488892 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-thanos" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488901 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="config-reloader" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488905 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="config-reloader" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488916 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="thanos-sidecar" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488920 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="thanos-sidecar" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488926 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488930 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488974 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488984 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="prometheus" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488991 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-web" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.488997 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="kube-rbac-proxy-thanos" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.489002 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="config-reloader" Nov 03 03:03:00.489026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.489008 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" containerName="thanos-sidecar" Nov 03 03:03:00.495546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.495503 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543447 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543411 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543447 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543450 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543473 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543527 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543570 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543668 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543691 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543715 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543757 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543795 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.543877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543866 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543908 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqw4\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-kube-api-access-cxqw4\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.543988 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544023 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544057 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544083 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544109 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544130 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544154 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.544357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.544184 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.550002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.549985 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 03:03:00.550085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.549996 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 03:03:00.550256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550240 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 03:03:00.550320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550275 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ba8kqj028vm5e\"" Nov 03 03:03:00.550366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550353 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cn4p\"" Nov 03 03:03:00.550681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550665 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 03:03:00.550754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550740 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 03:03:00.550801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.550762 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 03:03:00.551073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.551060 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 03:03:00.601334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601274 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 03:03:00.601334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601301 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 03:03:00.601334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601303 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 03:03:00.601334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601283 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 03:03:00.601617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601342 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 03:03:00.601617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601372 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 03:03:00.601617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601393 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 03:03:00.601617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.601412 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 03:03:00.602019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.602003 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:03:00.645265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645230 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqw4\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-kube-api-access-cxqw4\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645264 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645282 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645416 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645445 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645475 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645495 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645519 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645666 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.645761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645680 5275 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 03:03:00.646039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.645999 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646056 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646101 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646012 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646128 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646156 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.646158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646175 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646770 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646797 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.646831 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647006 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647041 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647076 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647105 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647466 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647594 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.647901 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.648669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.648179 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.649168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.649116 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.649270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.649178 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.649334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.649273 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.650010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.649969 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.650457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.650431 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.650809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.650700 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.650809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.650768 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25711346-0358-49b3-839c-fd65d3686ba2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.650974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.650804 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.651079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.651061 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25711346-0358-49b3-839c-fd65d3686ba2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.651276 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.651257 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-config\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.652029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.652010 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.652152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.652136 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25711346-0358-49b3-839c-fd65d3686ba2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.707124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.707092 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqw4\" (UniqueName: \"kubernetes.io/projected/25711346-0358-49b3-839c-fd65d3686ba2-kube-api-access-cxqw4\") pod \"prometheus-k8s-0\" (UID: \"25711346-0358-49b3-839c-fd65d3686ba2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.812762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.812720 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:00.982114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:00.982080 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f110ed-1cbf-478a-a45f-c2181a1a5563" path="/var/lib/kubelet/pods/29f110ed-1cbf-478a-a45f-c2181a1a5563/volumes" Nov 03 03:03:01.247301 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:03:01.247259 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25711346_0358_49b3_839c_fd65d3686ba2.slice/crio-44e7786babea5a22990f50146d9e09e615d2b8af1ca66f05f6d0b4820f4e33e5 WatchSource:0}: Error finding container 44e7786babea5a22990f50146d9e09e615d2b8af1ca66f05f6d0b4820f4e33e5: Status 404 returned error can't find the container with id 44e7786babea5a22990f50146d9e09e615d2b8af1ca66f05f6d0b4820f4e33e5 Nov 03 03:03:01.251444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:01.249346 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 03:03:01.285114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:01.285080 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"44e7786babea5a22990f50146d9e09e615d2b8af1ca66f05f6d0b4820f4e33e5"} Nov 03 03:03:02.291062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:02.291035 5275 generic.go:356] "Generic (PLEG): container finished" podID="25711346-0358-49b3-839c-fd65d3686ba2" containerID="748a6db11eed82c0fe66447e4bc799b96cc7a1251385f8172fb7e4c602367d53" exitCode=0 Nov 03 03:03:02.291456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:02.291097 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerDied","Data":"748a6db11eed82c0fe66447e4bc799b96cc7a1251385f8172fb7e4c602367d53"} Nov 03 03:03:02.330388 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:02.330354 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:03:03.297829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.297796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"f2e4f758fab0f1d2a57930ef7e4be07ce4e720e7ed0aa82f9aaa2002c4ff5b07"} Nov 03 03:03:03.298232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.297904 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"47d5c3f006d6d179a225bb83eb946c5252e953c5fc27e6b532ed1c00a9f1a75d"} Nov 03 03:03:03.298232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.297945 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"18895536e1c841ccb05e74fc80d5be07ea297c1bea9bf4550fc143845229aa17"} Nov 03 03:03:03.298232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.297983 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"87121b3484222a3868defdeae16d3dabb5cfec6f23832a15466a87bc7496b0d7"} Nov 03 03:03:03.298232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.298025 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"34cff427744f3383dcfa86314d1349253dd51ef7f9352980e5965e6202c548d1"} Nov 03 03:03:03.298232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.298059 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25711346-0358-49b3-839c-fd65d3686ba2","Type":"ContainerStarted","Data":"a272891a6904259cc722e09c10bd2a4e3cd7fcd782712639f62cf46e05ac9d12"} Nov 03 03:03:03.434088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:03.434025 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.416856673 podStartE2EDuration="3.43400707s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.150397ms" isStatefulPod=false podCreationTimestamp="2026-11-03 03:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:03:03.4332531 +0000 UTC m=+74.585941521" watchObservedRunningTime="2026-11-03 03:03:03.43400707 +0000 UTC m=+74.586695491" Nov 03 03:03:05.813573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:03:05.813526 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:03:32.328163 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:03:32.328079 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:04:00.813596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:04:00.813554 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:04:00.829627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:04:00.829600 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:04:01.513181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:04:01.513155 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 03:04:02.328380 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:04:02.328345 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:04:32.327248 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:04:32.327212 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:05:02.329028 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:05:02.328985 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:05:32.327250 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:05:32.327215 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:06:02.327985 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:06:02.327949 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:06:32.327819 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:06:32.327736 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:06:48.991369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:06:48.991342 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:06:48.993192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:06:48.993168 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:07:02.327585 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:07:02.327540 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:07:32.327438 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:07:32.327394 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:07:49.488432 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.488392 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t"] Nov 03 03:07:49.491083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.491047 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.592841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.592811 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 03:07:49.592841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.592818 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 03:07:49.593061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.592903 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-fr2jv\"" Nov 03 03:07:49.593459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.593437 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t"] Nov 03 03:07:49.603618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.603593 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5r9b\" (UniqueName: \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.603721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.603635 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.603794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.603763 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.704510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.704470 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.704686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.704525 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5r9b\" (UniqueName: \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.704686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.704559 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.704909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.704887 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.704956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.704917 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.765975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.765898 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5r9b\" (UniqueName: \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:49.803725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:49.803689 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:07:50.103261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.103228 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw"] Nov 03 03:07:50.105757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.105735 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.158445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.158411 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-fb4jt\"" Nov 03 03:07:50.165955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.165915 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw"] Nov 03 03:07:50.207014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.206978 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.207205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.207018 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxx28\" (UniqueName: \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.207205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.207092 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.217143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.217111 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t"] Nov 03 03:07:50.220624 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:50.220590 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0a9246_3970_468c_b567_379001efa81c.slice/crio-c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c WatchSource:0}: Error finding container c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c: Status 404 returned error can't find the container with id c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c Nov 03 03:07:50.224051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.224031 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:07:50.250087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.250047 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerStarted","Data":"c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c"} Nov 03 03:07:50.307476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.307435 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.307668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.307482 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx28\" (UniqueName: \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.307668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.307608 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.307897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.307880 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.307954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.307941 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.369403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.369335 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx28\" (UniqueName: \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.419617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.419582 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:07:50.790279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.790243 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw"] Nov 03 03:07:50.794089 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:50.794054 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b377df_462e_4180_b482_9a9d32eccae3.slice/crio-13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8 WatchSource:0}: Error finding container 13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8: Status 404 returned error can't find the container with id 13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8 Nov 03 03:07:50.871315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.871280 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd"] Nov 03 03:07:50.874587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.874571 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:50.926439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.926409 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-h2fkn\"" Nov 03 03:07:50.933355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:50.933325 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd"] Nov 03 03:07:51.012446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.012413 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf29\" (UniqueName: \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.012649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.012462 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.012649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.012498 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.013446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.013388 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs"] Nov 03 03:07:51.015622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.015605 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.069207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.069115 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lv77s\"" Nov 03 03:07:51.075729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.075699 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs"] Nov 03 03:07:51.106403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.106372 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l"] Nov 03 03:07:51.108677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.108661 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.113961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.113934 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf29\" (UniqueName: \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.114087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.113994 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.114087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114024 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.114087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114058 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.114259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114195 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.114317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114272 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzj2\" (UniqueName: \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.114438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114419 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.114540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.114522 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.169629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.169577 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2qbbf\"" Nov 03 03:07:51.170177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.170154 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l"] Nov 03 03:07:51.177570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.177543 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf29\" (UniqueName: \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.187304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.187271 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:07:51.201251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.201224 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2"] Nov 03 03:07:51.204327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.204283 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.214784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.214758 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.214897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.214806 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.214979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.214899 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.214979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.214935 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9wb\" (UniqueName: \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.214979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.214973 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.215146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.215053 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzj2\" (UniqueName: \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.215614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.215593 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.215711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.215661 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.255832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.255797 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerStarted","Data":"13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8"} Nov 03 03:07:51.263919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.263322 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2"] Nov 03 03:07:51.269381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.269356 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vm7qx\"" Nov 03 03:07:51.276594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.276567 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzj2\" (UniqueName: \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.304912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.304877 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8"] Nov 03 03:07:51.307629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.307605 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.316014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.315994 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.316127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316023 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.316127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316085 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9wb\" (UniqueName: \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.316127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316108 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.316245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316133 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.316245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316158 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bc4l\" (UniqueName: \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.316543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316526 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.316735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.316717 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.328245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.328181 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:07:51.369860 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.369802 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-622xh\"" Nov 03 03:07:51.371362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.371339 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8"] Nov 03 03:07:51.380013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.379787 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9wb\" (UniqueName: \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.417552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417522 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.417720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417561 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.417720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417581 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc4l\" (UniqueName: \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.417720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417702 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.417904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417881 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdw59\" (UniqueName: \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.417904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417898 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.418009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.417940 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.418009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.418002 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.421089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.421066 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:07:51.481863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.481817 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bc4l\" (UniqueName: \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.518938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.518903 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdw59\" (UniqueName: \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.519077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.518964 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.519077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.519009 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.519433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.519411 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.519473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.519420 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.521785 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.521764 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:07:51.573114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.573079 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd"] Nov 03 03:07:51.579532 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:51.579457 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13eaf12_eb81_44e6_8a07_bfa8ec7fd22f.slice/crio-262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b WatchSource:0}: Error finding container 262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b: Status 404 returned error can't find the container with id 262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b Nov 03 03:07:51.586543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.586515 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdw59\" (UniqueName: \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.623534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.623496 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:07:51.707035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.706940 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs"] Nov 03 03:07:51.711090 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:51.711059 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea4ae92_2855_4102_bdac_5e092d7e4f5f.slice/crio-f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123 WatchSource:0}: Error finding container f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123: Status 404 returned error can't find the container with id f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123 Nov 03 03:07:51.802824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.802785 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l"] Nov 03 03:07:51.807422 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:51.807390 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb14661c_879e_4e68_9e5b_678839c1f6eb.slice/crio-041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0 WatchSource:0}: Error finding container 041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0: Status 404 returned error can't find the container with id 041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0 Nov 03 03:07:51.903504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:51.903372 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2"] Nov 03 03:07:51.908052 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:51.908018 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931296c4_6660_40a5_9752_b5b4d9be457d.slice/crio-3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c WatchSource:0}: Error finding container 3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c: Status 404 returned error can't find the container with id 3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c Nov 03 03:07:52.009360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.009334 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8"] Nov 03 03:07:52.011707 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:52.011675 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8c0c54_7185_4120_9286_8d6a62cab5f7.slice/crio-fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5 WatchSource:0}: Error finding container fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5: Status 404 returned error can't find the container with id fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5 Nov 03 03:07:52.261088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.261002 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" event={"ID":"7ea4ae92-2855-4102-bdac-5e092d7e4f5f","Type":"ContainerStarted","Data":"f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123"} Nov 03 03:07:52.262416 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.262379 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" event={"ID":"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f","Type":"ContainerStarted","Data":"262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b"} Nov 03 03:07:52.263721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.263689 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerStarted","Data":"041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0"} Nov 03 03:07:52.264998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.264974 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerStarted","Data":"fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5"} Nov 03 03:07:52.266185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:52.266163 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" event={"ID":"931296c4-6660-40a5-9752-b5b4d9be457d","Type":"ContainerStarted","Data":"3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c"} Nov 03 03:07:54.804115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.804079 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k"] Nov 03 03:07:54.808729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.808054 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:54.912595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.912542 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k"] Nov 03 03:07:54.951455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.951412 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:54.951663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.951583 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7625\" (UniqueName: \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:54.951663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:54.951645 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.052268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.052230 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.052441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.052325 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7625\" (UniqueName: \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.052441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.052365 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.052836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.052805 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.053140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.053117 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.125419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.124617 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7625\" (UniqueName: \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.132269 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.132243 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:07:55.625777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:55.625695 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k"] Nov 03 03:07:55.654019 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:55.653982 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86897277_9722_48ab_a651_f0e4628319d3.slice/crio-f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d WatchSource:0}: Error finding container f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d: Status 404 returned error can't find the container with id f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d Nov 03 03:07:56.300089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:56.300002 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerStarted","Data":"f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d"} Nov 03 03:07:58.151782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.151743 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:07:58.156508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.155538 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204213 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204268 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204298 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204325 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204365 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204395 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204429 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204458 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.204923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.204493 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnpx\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.215557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.215505 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 03:07:58.216005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.215604 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 03:07:58.216005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.215673 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 03:07:58.218510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.218487 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\"" Nov 03 03:07:58.268143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.268043 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305707 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305744 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305777 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305805 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305859 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305893 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305927 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305952 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.305984 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnpx\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.306627 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.307322 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.307480 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.307643 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.307890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.307798 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.320653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.320598 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.324218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.324191 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.383932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.383833 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.387064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.387009 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnpx\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\") pod \"maas-default-gateway-openshift-default-76f4876667-cjnn9\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.476627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.476260 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:07:58.490943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.489105 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn"] Nov 03 03:07:58.493901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.493115 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507322 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507380 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507407 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507434 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507461 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b557j\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-kube-api-access-b557j\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507499 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507521 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d09bf2-ec59-49a4-8470-a956eefdded7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507550 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.507830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.507637 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.547253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.547222 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-28l4j\"" Nov 03 03:07:58.565798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.565765 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn"] Nov 03 03:07:58.608757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.608708 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.608924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.608784 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.608924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.608832 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609436 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609505 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609539 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609569 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b557j\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-kube-api-access-b557j\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609613 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609638 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d09bf2-ec59-49a4-8470-a956eefdded7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.609667 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.610008 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.610262 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.610934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.610878 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d09bf2-ec59-49a4-8470-a956eefdded7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.612070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.612026 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.618122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.618076 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.622439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.620094 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.679562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.679523 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b557j\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-kube-api-access-b557j\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.682795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.682765 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d09bf2-ec59-49a4-8470-a956eefdded7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn\" (UID: \"67d09bf2-ec59-49a4-8470-a956eefdded7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.812616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.812581 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:07:58.956514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:58.956417 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:07:59.240329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:59.240248 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn"] Nov 03 03:07:59.246754 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:07:59.246712 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d09bf2_ec59_49a4_8470_a956eefdded7.slice/crio-8a67503b9b00924ed91e88e2fe53869cc2d7280b63cd3eaf8a0e89827008c96e WatchSource:0}: Error finding container 8a67503b9b00924ed91e88e2fe53869cc2d7280b63cd3eaf8a0e89827008c96e: Status 404 returned error can't find the container with id 8a67503b9b00924ed91e88e2fe53869cc2d7280b63cd3eaf8a0e89827008c96e Nov 03 03:07:59.324032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:59.323543 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" event={"ID":"67d09bf2-ec59-49a4-8470-a956eefdded7","Type":"ContainerStarted","Data":"8a67503b9b00924ed91e88e2fe53869cc2d7280b63cd3eaf8a0e89827008c96e"} Nov 03 03:07:59.326972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:07:59.326263 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerStarted","Data":"4aa2572f079bfd62a177f042bc189109c47427918ec78f1ae4d19858881022c3"} Nov 03 03:08:02.369572 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:08:02.369465 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:08:09.380239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.380197 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.380624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.380294 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.380624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.380343 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.392544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.392507 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.392654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.392595 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.392654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.392634 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:08:09.457990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.456425 5275 generic.go:356] "Generic (PLEG): container finished" podID="82b377df-462e-4180-b482-9a9d32eccae3" containerID="bc599f9306ad8a4d3f4ba2b3b63d0e52998cc17843587a3f1ea1633540469da2" exitCode=0 Nov 03 03:08:09.457990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.457089 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerDied","Data":"bc599f9306ad8a4d3f4ba2b3b63d0e52998cc17843587a3f1ea1633540469da2"} Nov 03 03:08:09.460102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.459348 5275 generic.go:356] "Generic (PLEG): container finished" podID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerID="81a11a1534026342f1b477b2d5b5c486c9cccc93c2a50e92de339dd651ece48b" exitCode=0 Nov 03 03:08:09.460102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.459436 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" event={"ID":"7ea4ae92-2855-4102-bdac-5e092d7e4f5f","Type":"ContainerDied","Data":"81a11a1534026342f1b477b2d5b5c486c9cccc93c2a50e92de339dd651ece48b"} Nov 03 03:08:09.464475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.464449 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerStarted","Data":"00a82d55612f0625136f7a9d35608e80244e75cee2f3a77c972ad996edcbeadc"} Nov 03 03:08:09.467168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.467143 5275 generic.go:356] "Generic (PLEG): container finished" podID="7e0a9246-3970-468c-b567-379001efa81c" containerID="f520073d11bcfd17310d52633195baab8433013c3f5a03f7ff951c2a68d5c98d" exitCode=0 Nov 03 03:08:09.472869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.472712 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerDied","Data":"f520073d11bcfd17310d52633195baab8433013c3f5a03f7ff951c2a68d5c98d"} Nov 03 03:08:09.486665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.486578 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerStarted","Data":"05d15104d25b176af8d1d11eab279ffa8272ce67cc26caae3ae53c91d34367bb"} Nov 03 03:08:09.490118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:09.489555 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerStarted","Data":"2d1bb8c064ec71566e228a48df9c1ef1c8213a6bec2c67767cc291eb90f9ecf2"} Nov 03 03:08:10.495328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.495289 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerStarted","Data":"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb"} Nov 03 03:08:10.496820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.496791 5275 generic.go:356] "Generic (PLEG): container finished" podID="931296c4-6660-40a5-9752-b5b4d9be457d" containerID="0a1fa7cf0ce37ed98d847b4830c4c9b14179ec25560cf4fcb9c0d7296bf74fae" exitCode=0 Nov 03 03:08:10.496941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.496879 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" event={"ID":"931296c4-6660-40a5-9752-b5b4d9be457d","Type":"ContainerDied","Data":"0a1fa7cf0ce37ed98d847b4830c4c9b14179ec25560cf4fcb9c0d7296bf74fae"} Nov 03 03:08:10.498382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.498361 5275 generic.go:356] "Generic (PLEG): container finished" podID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerID="afece5388a8537826d562a8f04516a6d08e4e36985b3c7d9af908795bd9baa72" exitCode=0 Nov 03 03:08:10.498476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.498443 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" event={"ID":"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f","Type":"ContainerDied","Data":"afece5388a8537826d562a8f04516a6d08e4e36985b3c7d9af908795bd9baa72"} Nov 03 03:08:10.500033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.500011 5275 generic.go:356] "Generic (PLEG): container finished" podID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerID="00a82d55612f0625136f7a9d35608e80244e75cee2f3a77c972ad996edcbeadc" exitCode=0 Nov 03 03:08:10.500121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.500107 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerDied","Data":"00a82d55612f0625136f7a9d35608e80244e75cee2f3a77c972ad996edcbeadc"} Nov 03 03:08:10.502054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.502031 5275 generic.go:356] "Generic (PLEG): container finished" podID="86897277-9722-48ab-a651-f0e4628319d3" containerID="05d15104d25b176af8d1d11eab279ffa8272ce67cc26caae3ae53c91d34367bb" exitCode=0 Nov 03 03:08:10.502150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.502101 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerDied","Data":"05d15104d25b176af8d1d11eab279ffa8272ce67cc26caae3ae53c91d34367bb"} Nov 03 03:08:10.503813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.503752 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" event={"ID":"67d09bf2-ec59-49a4-8470-a956eefdded7","Type":"ContainerStarted","Data":"d0255746e6f120b32bd9292481ad13e052391e57ce59519238a5ca300713e43a"} Nov 03 03:08:10.505273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.505251 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerID="2d1bb8c064ec71566e228a48df9c1ef1c8213a6bec2c67767cc291eb90f9ecf2" exitCode=0 Nov 03 03:08:10.505357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.505282 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerDied","Data":"2d1bb8c064ec71566e228a48df9c1ef1c8213a6bec2c67767cc291eb90f9ecf2"} Nov 03 03:08:10.616635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.616577 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podStartSLOduration=2.206094363 podStartE2EDuration="12.616561656s" totalImagesPullingTime="10.410467293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:10.615332927 +0000 UTC m=+381.768021348" watchObservedRunningTime="2026-11-03 03:08:10.616561656 +0000 UTC m=+381.769250076" Nov 03 03:08:10.813604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.813567 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:08:10.818480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.818454 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:08:10.982091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:10.982041 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" podStartSLOduration=2.8397075210000002 podStartE2EDuration="12.982025147s" totalImagesPullingTime="10.142317626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:10.980695832 +0000 UTC m=+382.133384252" watchObservedRunningTime="2026-11-03 03:08:10.982025147 +0000 UTC m=+382.134713571" Nov 03 03:08:11.476775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.476732 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:08:11.482422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.482378 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:08:11.514180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.514150 5275 generic.go:356] "Generic (PLEG): container finished" podID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerID="3ffb5eb3ae7354f5701ee4fbafd8fccaf056e3bd840b1938be0241d96ede8169" exitCode=0 Nov 03 03:08:11.514560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.514245 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" event={"ID":"7ea4ae92-2855-4102-bdac-5e092d7e4f5f","Type":"ContainerDied","Data":"3ffb5eb3ae7354f5701ee4fbafd8fccaf056e3bd840b1938be0241d96ede8169"} Nov 03 03:08:11.517427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.517400 5275 generic.go:356] "Generic (PLEG): container finished" podID="7e0a9246-3970-468c-b567-379001efa81c" containerID="334d3311297c8508e703584e7b82e8c3901a389daaea5db285a7b11de5be9769" exitCode=0 Nov 03 03:08:11.517556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.517487 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerDied","Data":"334d3311297c8508e703584e7b82e8c3901a389daaea5db285a7b11de5be9769"} Nov 03 03:08:11.519511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.519490 5275 generic.go:356] "Generic (PLEG): container finished" podID="86897277-9722-48ab-a651-f0e4628319d3" containerID="12079e93dea7214b0482658323e7a7da9f753373cb42cdca700c581794d5e510" exitCode=0 Nov 03 03:08:11.519587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.519568 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerDied","Data":"12079e93dea7214b0482658323e7a7da9f753373cb42cdca700c581794d5e510"} Nov 03 03:08:11.520067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.520050 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:08:11.520173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.520092 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:08:11.521135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.521113 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:08:11.521235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:11.521173 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn" Nov 03 03:08:12.527863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.527811 5275 generic.go:356] "Generic (PLEG): container finished" podID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerID="e552dad1829cf420137a7ac630d26110c387401d812c94e268565c7efe49ebae" exitCode=0 Nov 03 03:08:12.528310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.527956 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerDied","Data":"e552dad1829cf420137a7ac630d26110c387401d812c94e268565c7efe49ebae"} Nov 03 03:08:12.531562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.531537 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerID="8946afec2b1bb79767df38bbe2d653cb4b090ae2d721e3a53c47268c0740f285" exitCode=0 Nov 03 03:08:12.531806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.531620 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerDied","Data":"8946afec2b1bb79767df38bbe2d653cb4b090ae2d721e3a53c47268c0740f285"} Nov 03 03:08:12.535170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.535148 5275 generic.go:356] "Generic (PLEG): container finished" podID="931296c4-6660-40a5-9752-b5b4d9be457d" containerID="5418d593d04598ff4036380d5d6c9c021530f9b5f54d11ca704ca0b3e7604f2f" exitCode=0 Nov 03 03:08:12.535293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:12.535225 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" event={"ID":"931296c4-6660-40a5-9752-b5b4d9be457d","Type":"ContainerDied","Data":"5418d593d04598ff4036380d5d6c9c021530f9b5f54d11ca704ca0b3e7604f2f"} Nov 03 03:08:13.544270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:13.544196 5275 generic.go:356] "Generic (PLEG): container finished" podID="82b377df-462e-4180-b482-9a9d32eccae3" containerID="2aa9bf8aa982dd8d84677615de039ddc7f6c24166a601c8b02a10e4be4b89738" exitCode=0 Nov 03 03:08:13.545472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:13.545444 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerDied","Data":"2aa9bf8aa982dd8d84677615de039ddc7f6c24166a601c8b02a10e4be4b89738"} Nov 03 03:08:16.582681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:16.582521 5275 generic.go:356] "Generic (PLEG): container finished" podID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerID="bb0d17ba0703f72a2cf7bd40d61c31aced17183141a6f00dfe69bcfb841f4adc" exitCode=0 Nov 03 03:08:16.582681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:16.582625 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" event={"ID":"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f","Type":"ContainerDied","Data":"bb0d17ba0703f72a2cf7bd40d61c31aced17183141a6f00dfe69bcfb841f4adc"} Nov 03 03:08:29.718993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:29.718059 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerStarted","Data":"a6b4d3526b606b5e9d452433c428b5d55b5c25b783d5bfc5ecea2344db3e0912"} Nov 03 03:08:29.723708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:29.723417 5275 generic.go:356] "Generic (PLEG): container finished" podID="86897277-9722-48ab-a651-f0e4628319d3" containerID="3c8dc367372e9e175f9d2861c75be5968934ad4d8d71901a88fc0f68288e9e09" exitCode=0 Nov 03 03:08:29.723708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:29.723494 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerDied","Data":"3c8dc367372e9e175f9d2861c75be5968934ad4d8d71901a88fc0f68288e9e09"} Nov 03 03:08:29.842448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:29.842379 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" podStartSLOduration=4.215361529 podStartE2EDuration="40.842359335s" totalImagesPullingTime="36.626997806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:29.840766212 +0000 UTC m=+400.993454632" watchObservedRunningTime="2026-11-03 03:08:29.842359335 +0000 UTC m=+400.995047756" Nov 03 03:08:30.732695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.732660 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerStarted","Data":"9196cf3e5b6e4ebe900ab2aa2726bb7d8dc45c93a9e34d5b137ccc867870c505"} Nov 03 03:08:30.736351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.736317 5275 generic.go:356] "Generic (PLEG): container finished" podID="7e0a9246-3970-468c-b567-379001efa81c" containerID="a6b4d3526b606b5e9d452433c428b5d55b5c25b783d5bfc5ecea2344db3e0912" exitCode=0 Nov 03 03:08:30.736589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.736566 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerDied","Data":"a6b4d3526b606b5e9d452433c428b5d55b5c25b783d5bfc5ecea2344db3e0912"} Nov 03 03:08:30.739194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.739163 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerID="c58b876d90ef2d0952e2787d0d0ba5b34872b0f139c3958d1a44047b65bd927a" exitCode=0 Nov 03 03:08:30.739324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.739257 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerDied","Data":"c58b876d90ef2d0952e2787d0d0ba5b34872b0f139c3958d1a44047b65bd927a"} Nov 03 03:08:30.742069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.742042 5275 generic.go:356] "Generic (PLEG): container finished" podID="931296c4-6660-40a5-9752-b5b4d9be457d" containerID="b9539f21263998e03bb71f9b5ee42e54151ec67f1906cb769c79507b16f4b0ba" exitCode=0 Nov 03 03:08:30.742196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.742168 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" event={"ID":"931296c4-6660-40a5-9752-b5b4d9be457d","Type":"ContainerDied","Data":"b9539f21263998e03bb71f9b5ee42e54151ec67f1906cb769c79507b16f4b0ba"} Nov 03 03:08:30.745055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.745028 5275 generic.go:356] "Generic (PLEG): container finished" podID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerID="668958562b460a0cfab96fe4808ece0674535852b362c35e03115b7978715399" exitCode=0 Nov 03 03:08:30.745201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.745151 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" event={"ID":"7ea4ae92-2855-4102-bdac-5e092d7e4f5f","Type":"ContainerDied","Data":"668958562b460a0cfab96fe4808ece0674535852b362c35e03115b7978715399"} Nov 03 03:08:30.913666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.913641 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:08:30.978207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:30.978155 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" podStartSLOduration=3.371829995 podStartE2EDuration="39.978139057s" totalImagesPullingTime="36.606309062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:30.858016294 +0000 UTC m=+402.010704716" watchObservedRunningTime="2026-11-03 03:08:30.978139057 +0000 UTC m=+402.130827478" Nov 03 03:08:31.046239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.046203 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\" (UniqueName: \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\") pod \"86897277-9722-48ab-a651-f0e4628319d3\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " Nov 03 03:08:31.046433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.046257 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\") pod \"86897277-9722-48ab-a651-f0e4628319d3\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " Nov 03 03:08:31.046433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.046357 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\") pod \"86897277-9722-48ab-a651-f0e4628319d3\" (UID: \"86897277-9722-48ab-a651-f0e4628319d3\") " Nov 03 03:08:31.046825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.046799 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle" pod "86897277-9722-48ab-a651-f0e4628319d3" (UID: "86897277-9722-48ab-a651-f0e4628319d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:31.048800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.048735 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625" pod "86897277-9722-48ab-a651-f0e4628319d3" (UID: "86897277-9722-48ab-a651-f0e4628319d3"). InnerVolumeSpecName "kube-api-access-h7625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:31.053901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.053870 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util" pod "86897277-9722-48ab-a651-f0e4628319d3" (UID: "86897277-9722-48ab-a651-f0e4628319d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:31.147979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.147941 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:31.147979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.147976 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7625\" (UniqueName: \"kubernetes.io/projected/86897277-9722-48ab-a651-f0e4628319d3-kube-api-access-h7625\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:31.148226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.148014 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86897277-9722-48ab-a651-f0e4628319d3-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:31.753186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.753137 5275 generic.go:356] "Generic (PLEG): container finished" podID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerID="9196cf3e5b6e4ebe900ab2aa2726bb7d8dc45c93a9e34d5b137ccc867870c505" exitCode=0 Nov 03 03:08:31.753623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.753282 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerDied","Data":"9196cf3e5b6e4ebe900ab2aa2726bb7d8dc45c93a9e34d5b137ccc867870c505"} Nov 03 03:08:31.755417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.755395 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" Nov 03 03:08:31.755556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.755417 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71h4n6k" event={"ID":"86897277-9722-48ab-a651-f0e4628319d3","Type":"ContainerDied","Data":"f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d"} Nov 03 03:08:31.755556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.755464 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31abb61e3e3b2c21e86bd9553d1c67717fc6a8a1bc649d8124cd4d74d4b729d" Nov 03 03:08:31.757711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.757644 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerStarted","Data":"0047e087dd88d5e8d80cc7de99c2193fbaadc7bd007b10a23c94af8702f22a82"} Nov 03 03:08:31.950492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.950389 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:08:31.973047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.972963 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:08:31.975810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.975789 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:08:31.979649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:31.979631 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:08:32.056352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056318 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\" (UniqueName: \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\") pod \"7e0a9246-3970-468c-b567-379001efa81c\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " Nov 03 03:08:32.056539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056363 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\" (UniqueName: \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\") pod \"931296c4-6660-40a5-9752-b5b4d9be457d\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " Nov 03 03:08:32.056539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056456 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\") pod \"7e0a9246-3970-468c-b567-379001efa81c\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " Nov 03 03:08:32.056539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056487 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\") pod \"7e0a9246-3970-468c-b567-379001efa81c\" (UID: \"7e0a9246-3970-468c-b567-379001efa81c\") " Nov 03 03:08:32.056539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056503 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\") pod \"931296c4-6660-40a5-9752-b5b4d9be457d\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " Nov 03 03:08:32.056750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.056540 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\") pod \"931296c4-6660-40a5-9752-b5b4d9be457d\" (UID: \"931296c4-6660-40a5-9752-b5b4d9be457d\") " Nov 03 03:08:32.057140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.057117 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle" pod "7e0a9246-3970-468c-b567-379001efa81c" (UID: "7e0a9246-3970-468c-b567-379001efa81c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.057492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.057464 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle" pod "931296c4-6660-40a5-9752-b5b4d9be457d" (UID: "931296c4-6660-40a5-9752-b5b4d9be457d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.058891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.058864 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b" pod "7e0a9246-3970-468c-b567-379001efa81c" (UID: "7e0a9246-3970-468c-b567-379001efa81c"). InnerVolumeSpecName "kube-api-access-r5r9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:32.059288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.059263 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l" pod "931296c4-6660-40a5-9752-b5b4d9be457d" (UID: "931296c4-6660-40a5-9752-b5b4d9be457d"). InnerVolumeSpecName "kube-api-access-2bc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:32.060417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.059886 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" podStartSLOduration=2.057898501 podStartE2EDuration="42.059869109s" totalImagesPullingTime="40.001970608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:32.058782479 +0000 UTC m=+403.211470899" watchObservedRunningTime="2026-11-03 03:08:32.059869109 +0000 UTC m=+403.212557530" Nov 03 03:08:32.063250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.063224 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util" pod "7e0a9246-3970-468c-b567-379001efa81c" (UID: "7e0a9246-3970-468c-b567-379001efa81c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.063827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.063804 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util" pod "931296c4-6660-40a5-9752-b5b4d9be457d" (UID: "931296c4-6660-40a5-9752-b5b4d9be457d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.157381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157292 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\" (UniqueName: \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\") pod \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " Nov 03 03:08:32.157533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157400 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\") pod \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " Nov 03 03:08:32.157533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157461 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\" (UniqueName: \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\") pod \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " Nov 03 03:08:32.157533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157491 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\") pod \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " Nov 03 03:08:32.157697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157547 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\") pod \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\" (UID: \"7ea4ae92-2855-4102-bdac-5e092d7e4f5f\") " Nov 03 03:08:32.157697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157574 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\") pod \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\" (UID: \"5c8c0c54-7185-4120-9286-8d6a62cab5f7\") " Nov 03 03:08:32.157841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157824 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.157941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157885 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e0a9246-3970-468c-b567-379001efa81c-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.157941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157900 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.157941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157913 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/931296c4-6660-40a5-9752-b5b4d9be457d-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.157941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157926 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5r9b\" (UniqueName: \"kubernetes.io/projected/7e0a9246-3970-468c-b567-379001efa81c-kube-api-access-r5r9b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.158100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.157941 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bc4l\" (UniqueName: \"kubernetes.io/projected/931296c4-6660-40a5-9752-b5b4d9be457d-kube-api-access-2bc4l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.158100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.158001 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle" pod "7ea4ae92-2855-4102-bdac-5e092d7e4f5f" (UID: "7ea4ae92-2855-4102-bdac-5e092d7e4f5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.158172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.158142 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle" pod "5c8c0c54-7185-4120-9286-8d6a62cab5f7" (UID: "5c8c0c54-7185-4120-9286-8d6a62cab5f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.160109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.160084 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2" pod "7ea4ae92-2855-4102-bdac-5e092d7e4f5f" (UID: "7ea4ae92-2855-4102-bdac-5e092d7e4f5f"). InnerVolumeSpecName "kube-api-access-4bzj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:32.160490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.160453 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59" pod "5c8c0c54-7185-4120-9286-8d6a62cab5f7" (UID: "5c8c0c54-7185-4120-9286-8d6a62cab5f7"). InnerVolumeSpecName "kube-api-access-sdw59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:32.165508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.165160 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util" pod "7ea4ae92-2855-4102-bdac-5e092d7e4f5f" (UID: "7ea4ae92-2855-4102-bdac-5e092d7e4f5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.165508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.165446 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util" pod "5c8c0c54-7185-4120-9286-8d6a62cab5f7" (UID: "5c8c0c54-7185-4120-9286-8d6a62cab5f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:32.259030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.258993 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdw59\" (UniqueName: \"kubernetes.io/projected/5c8c0c54-7185-4120-9286-8d6a62cab5f7-kube-api-access-sdw59\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.259188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.259037 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.259188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.259054 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzj2\" (UniqueName: \"kubernetes.io/projected/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-kube-api-access-4bzj2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.259188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.259067 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.259188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.259083 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ea4ae92-2855-4102-bdac-5e092d7e4f5f-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.259188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.259095 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8c0c54-7185-4120-9286-8d6a62cab5f7-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:32.327893 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:08:32.327841 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:08:32.764423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.764381 5275 generic.go:356] "Generic (PLEG): container finished" podID="82b377df-462e-4180-b482-9a9d32eccae3" containerID="0047e087dd88d5e8d80cc7de99c2193fbaadc7bd007b10a23c94af8702f22a82" exitCode=0 Nov 03 03:08:32.764864 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.764458 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerDied","Data":"0047e087dd88d5e8d80cc7de99c2193fbaadc7bd007b10a23c94af8702f22a82"} Nov 03 03:08:32.766351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.766325 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" event={"ID":"7ea4ae92-2855-4102-bdac-5e092d7e4f5f","Type":"ContainerDied","Data":"f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123"} Nov 03 03:08:32.766461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.766372 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f519341b2cc2394bb9fbe797c97d4f3ad6df5200550cd4706c2cd8a76b845123" Nov 03 03:08:32.766461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.766334 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vhxqs" Nov 03 03:08:32.768613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.768582 5275 generic.go:356] "Generic (PLEG): container finished" podID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerID="a05112309c1071945737ea18af01133d1328dd2a904f93cc723a613d4736bac7" exitCode=0 Nov 03 03:08:32.768716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.768630 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" event={"ID":"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f","Type":"ContainerDied","Data":"a05112309c1071945737ea18af01133d1328dd2a904f93cc723a613d4736bac7"} Nov 03 03:08:32.770548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.770525 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" event={"ID":"7e0a9246-3970-468c-b567-379001efa81c","Type":"ContainerDied","Data":"c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c"} Nov 03 03:08:32.770548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.770539 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cmt7t" Nov 03 03:08:32.770671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.770564 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c28c4c69480b451682a3bcbd3e2ee5962feca3a1d1518ee673e9d79eaac8d93c" Nov 03 03:08:32.772496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.772473 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" event={"ID":"5c8c0c54-7185-4120-9286-8d6a62cab5f7","Type":"ContainerDied","Data":"fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5"} Nov 03 03:08:32.772496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.772489 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq74w8" Nov 03 03:08:32.772693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.772508 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe773a0abac7ddb520f3efc224188da83aaaee1188f25c5b00cee061d638a1d5" Nov 03 03:08:32.774582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.774559 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" event={"ID":"931296c4-6660-40a5-9752-b5b4d9be457d","Type":"ContainerDied","Data":"3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c"} Nov 03 03:08:32.774684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.774596 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb765381e81b0f2e62a77938a9227494988c0210de7cf74d330990052f0519c" Nov 03 03:08:32.774750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.774739 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a87hg2" Nov 03 03:08:32.903993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:32.903968 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:08:33.066084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.066050 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\" (UniqueName: \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\") pod \"fb14661c-879e-4e68-9e5b-678839c1f6eb\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " Nov 03 03:08:33.066287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.066121 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\") pod \"fb14661c-879e-4e68-9e5b-678839c1f6eb\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " Nov 03 03:08:33.066287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.066180 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\") pod \"fb14661c-879e-4e68-9e5b-678839c1f6eb\" (UID: \"fb14661c-879e-4e68-9e5b-678839c1f6eb\") " Nov 03 03:08:33.066716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.066682 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle" pod "fb14661c-879e-4e68-9e5b-678839c1f6eb" (UID: "fb14661c-879e-4e68-9e5b-678839c1f6eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:33.068397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.068373 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb" pod "fb14661c-879e-4e68-9e5b-678839c1f6eb" (UID: "fb14661c-879e-4e68-9e5b-678839c1f6eb"). InnerVolumeSpecName "kube-api-access-gt9wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:33.071889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.071863 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util" pod "fb14661c-879e-4e68-9e5b-678839c1f6eb" (UID: "fb14661c-879e-4e68-9e5b-678839c1f6eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:33.167254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.167217 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:33.167254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.167244 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb14661c-879e-4e68-9e5b-678839c1f6eb-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:33.167254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.167259 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9wb\" (UniqueName: \"kubernetes.io/projected/fb14661c-879e-4e68-9e5b-678839c1f6eb-kube-api-access-gt9wb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:33.780927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.780896 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" Nov 03 03:08:33.781440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.780897 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kmn9l" event={"ID":"fb14661c-879e-4e68-9e5b-678839c1f6eb","Type":"ContainerDied","Data":"041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0"} Nov 03 03:08:33.781440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.781028 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="041bf25333ae575e0de57d5c018cf3d667420a830302f83dfddc783ffbd985a0" Nov 03 03:08:33.904318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.904293 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:08:33.933458 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:33.933434 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:08:34.075339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075245 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\" (UniqueName: \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\") pod \"82b377df-462e-4180-b482-9a9d32eccae3\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " Nov 03 03:08:34.075339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075293 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\") pod \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " Nov 03 03:08:34.075339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075316 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\") pod \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " Nov 03 03:08:34.075577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075352 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\") pod \"82b377df-462e-4180-b482-9a9d32eccae3\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " Nov 03 03:08:34.075577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075410 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\") pod \"82b377df-462e-4180-b482-9a9d32eccae3\" (UID: \"82b377df-462e-4180-b482-9a9d32eccae3\") " Nov 03 03:08:34.075577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075454 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\" (UniqueName: \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\") pod \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\" (UID: \"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f\") " Nov 03 03:08:34.075840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.075807 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle" pod "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" (UID: "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.076520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.076491 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle" pod "82b377df-462e-4180-b482-9a9d32eccae3" (UID: "82b377df-462e-4180-b482-9a9d32eccae3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.077685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.077663 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28" pod "82b377df-462e-4180-b482-9a9d32eccae3" (UID: "82b377df-462e-4180-b482-9a9d32eccae3"). InnerVolumeSpecName "kube-api-access-fxx28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.077784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.077768 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29" pod "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" (UID: "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f"). InnerVolumeSpecName "kube-api-access-mmf29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.081603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.081583 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util" pod "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" (UID: "c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.083649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.083429 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util" pod "82b377df-462e-4180-b482-9a9d32eccae3" (UID: "82b377df-462e-4180-b482-9a9d32eccae3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.176017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.175990 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.176017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.176015 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.176017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.176024 5275 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-util\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.176223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.176033 5275 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82b377df-462e-4180-b482-9a9d32eccae3-bundle\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.176223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.176043 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf29\" (UniqueName: \"kubernetes.io/projected/c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f-kube-api-access-mmf29\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.176223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.176052 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxx28\" (UniqueName: \"kubernetes.io/projected/82b377df-462e-4180-b482-9a9d32eccae3-kube-api-access-fxx28\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.787376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.787346 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" Nov 03 03:08:34.787376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.787346 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpxlgw" event={"ID":"82b377df-462e-4180-b482-9a9d32eccae3","Type":"ContainerDied","Data":"13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8"} Nov 03 03:08:34.787892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.787397 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13dd98cb9cd086d8fd2bbc23cd067d2f9cabe6f665728dbc77b5dc68ca49b9d8" Nov 03 03:08:34.789304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.789280 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" event={"ID":"c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f","Type":"ContainerDied","Data":"262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b"} Nov 03 03:08:34.789417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.789320 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262780593e3fd34e735555840327ddcdb759452f3ed81ac6fdf6f5d031ef0f7b" Nov 03 03:08:34.789417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.789288 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pfdqd" Nov 03 03:08:34.947706 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.947676 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw"] Nov 03 03:08:34.948001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.947988 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="extract" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948003 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="extract" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948013 5275 cpu_manager.go:414] "containerMap: removing container" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="extract" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948018 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="extract" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948027 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="util" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948033 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="util" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948044 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="util" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948049 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="util" Nov 03 03:08:34.948055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948055 5275 cpu_manager.go:414] "containerMap: removing container" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948060 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948066 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948070 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948081 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948086 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948093 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948097 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948104 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948108 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948114 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948119 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948125 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948129 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948136 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948140 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948146 5275 cpu_manager.go:414] "containerMap: removing container" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948150 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948154 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948158 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948165 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948169 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948176 5275 cpu_manager.go:414] "containerMap: removing container" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948186 5275 cpu_manager.go:414] "containerMap: removing container" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948189 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948197 5275 cpu_manager.go:414] "containerMap: removing container" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948202 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948208 5275 cpu_manager.go:414] "containerMap: removing container" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948212 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="util" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948218 5275 cpu_manager.go:414] "containerMap: removing container" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948222 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948227 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948231 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="extract" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948241 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948245 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948251 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948255 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948260 5275 cpu_manager.go:414] "containerMap: removing container" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="pull" Nov 03 03:08:34.948283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948264 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="pull" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948303 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13eaf12-eb81-44e6-8a07-bfa8ec7fd22f" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948310 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0a9246-3970-468c-b567-379001efa81c" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948316 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea4ae92-2855-4102-bdac-5e092d7e4f5f" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948323 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="86897277-9722-48ab-a651-f0e4628319d3" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948329 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="931296c4-6660-40a5-9752-b5b4d9be457d" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948334 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8c0c54-7185-4120-9286-8d6a62cab5f7" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948340 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb14661c-879e-4e68-9e5b-678839c1f6eb" containerName="extract" Nov 03 03:08:34.949352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.948345 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b377df-462e-4180-b482-9a9d32eccae3" containerName="extract" Nov 03 03:08:34.950675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:34.950658 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:35.009240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.009214 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 03:08:35.009414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.009245 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r8sjz\"" Nov 03 03:08:35.009414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.009285 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 03:08:35.012688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.012650 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw"] Nov 03 03:08:35.084692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.084603 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrtr\" (UniqueName: \"kubernetes.io/projected/a60f9acc-e4f3-46e1-9951-f0fb96818e0d-kube-api-access-nvrtr\") pod \"limitador-operator-controller-manager-5778bb4449-z9kkw\" (UID: \"a60f9acc-e4f3-46e1-9951-f0fb96818e0d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:35.185662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.185626 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrtr\" (UniqueName: \"kubernetes.io/projected/a60f9acc-e4f3-46e1-9951-f0fb96818e0d-kube-api-access-nvrtr\") pod \"limitador-operator-controller-manager-5778bb4449-z9kkw\" (UID: \"a60f9acc-e4f3-46e1-9951-f0fb96818e0d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:35.300654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.300624 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrtr\" (UniqueName: \"kubernetes.io/projected/a60f9acc-e4f3-46e1-9951-f0fb96818e0d-kube-api-access-nvrtr\") pod \"limitador-operator-controller-manager-5778bb4449-z9kkw\" (UID: \"a60f9acc-e4f3-46e1-9951-f0fb96818e0d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:35.562015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.561971 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:35.949592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:35.949520 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw"] Nov 03 03:08:35.952514 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:08:35.952486 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60f9acc_e4f3_46e1_9951_f0fb96818e0d.slice/crio-ddd49bb794d49777c11b1db94d4927994bf487d7a8cdae4831aa7f8fa8bbc21e WatchSource:0}: Error finding container ddd49bb794d49777c11b1db94d4927994bf487d7a8cdae4831aa7f8fa8bbc21e: Status 404 returned error can't find the container with id ddd49bb794d49777c11b1db94d4927994bf487d7a8cdae4831aa7f8fa8bbc21e Nov 03 03:08:36.800030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:36.799986 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" event={"ID":"a60f9acc-e4f3-46e1-9951-f0fb96818e0d","Type":"ContainerStarted","Data":"ddd49bb794d49777c11b1db94d4927994bf487d7a8cdae4831aa7f8fa8bbc21e"} Nov 03 03:08:40.049798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.049759 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj"] Nov 03 03:08:40.052461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.052437 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.113553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.113459 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 03:08:40.137341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137303 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.137506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137372 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlbm\" (UniqueName: \"kubernetes.io/projected/75757f51-58f6-44f7-8a55-7ccd7b82ee28-kube-api-access-sxlbm\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.137506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137443 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.137506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137481 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.137615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137602 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.137659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.137631 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.164232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.163259 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-slhgs\"" Nov 03 03:08:40.164232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.163342 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 03:08:40.164232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.163556 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 03:08:40.164232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.163637 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 03:08:40.164232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.163720 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 03:08:40.166300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.166268 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj"] Nov 03 03:08:40.238181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238145 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238194 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238248 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238270 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlbm\" (UniqueName: \"kubernetes.io/projected/75757f51-58f6-44f7-8a55-7ccd7b82ee28-kube-api-access-sxlbm\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238303 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238343 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.238748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.238716 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.240802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.240776 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75757f51-58f6-44f7-8a55-7ccd7b82ee28-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.240954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.240930 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.241023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.240963 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.241080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.241060 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75757f51-58f6-44f7-8a55-7ccd7b82ee28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.303007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.302928 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlbm\" (UniqueName: \"kubernetes.io/projected/75757f51-58f6-44f7-8a55-7ccd7b82ee28-kube-api-access-sxlbm\") pod \"opendatahub-operator-controller-manager-69c4cfd777-swhwj\" (UID: \"75757f51-58f6-44f7-8a55-7ccd7b82ee28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.370032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.369992 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:40.825124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.825091 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" event={"ID":"a60f9acc-e4f3-46e1-9951-f0fb96818e0d","Type":"ContainerStarted","Data":"361d487962e70e38aa88d2a82ad18935aa0427d26ec878c4a4c15b6941176d2e"} Nov 03 03:08:40.825290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.825245 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:40.918918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.918877 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj"] Nov 03 03:08:40.924310 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:08:40.924279 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75757f51_58f6_44f7_8a55_7ccd7b82ee28.slice/crio-769885f7c5c58cb5b397ad0eff5dbdbbd7421193c02b134ffa28eaeec727ef86 WatchSource:0}: Error finding container 769885f7c5c58cb5b397ad0eff5dbdbbd7421193c02b134ffa28eaeec727ef86: Status 404 returned error can't find the container with id 769885f7c5c58cb5b397ad0eff5dbdbbd7421193c02b134ffa28eaeec727ef86 Nov 03 03:08:40.956647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:40.956593 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" podStartSLOduration=2.76331487 podStartE2EDuration="6.956579748s" totalImagesPullingTime="4.193264878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:40.954956449 +0000 UTC m=+412.107644867" watchObservedRunningTime="2026-11-03 03:08:40.956579748 +0000 UTC m=+412.109268166" Nov 03 03:08:41.838784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:41.838747 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" event={"ID":"75757f51-58f6-44f7-8a55-7ccd7b82ee28","Type":"ContainerStarted","Data":"769885f7c5c58cb5b397ad0eff5dbdbbd7421193c02b134ffa28eaeec727ef86"} Nov 03 03:08:42.923545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:42.923508 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 03:08:42.926086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:42.926065 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:42.962318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:42.962091 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgpl\" (UniqueName: \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:42.962318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:42.962215 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:42.984566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:42.984534 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 03:08:43.028947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.028913 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q9rn7\"" Nov 03 03:08:43.063928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.063444 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:43.063928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.063532 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgpl\" (UniqueName: \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:43.063928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.063927 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:43.139015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.138988 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgpl\" (UniqueName: \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-nvhkn\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:43.240057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.239966 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:08:43.731464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.731434 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 03:08:43.734127 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:08:43.734093 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f0dd9b_eedd_4bbd_99d1_f422d2e9fc04.slice/crio-ae4d49b6cd33d5ffcd7b655cb9138349e00bbc59317706ce7a04b7d2935e6976 WatchSource:0}: Error finding container ae4d49b6cd33d5ffcd7b655cb9138349e00bbc59317706ce7a04b7d2935e6976: Status 404 returned error can't find the container with id ae4d49b6cd33d5ffcd7b655cb9138349e00bbc59317706ce7a04b7d2935e6976 Nov 03 03:08:43.853480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:43.853438 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" event={"ID":"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04","Type":"ContainerStarted","Data":"ae4d49b6cd33d5ffcd7b655cb9138349e00bbc59317706ce7a04b7d2935e6976"} Nov 03 03:08:45.864637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:45.864603 5275 generic.go:356] "Generic (PLEG): container finished" podID="75757f51-58f6-44f7-8a55-7ccd7b82ee28" containerID="0b825a3565728dec730ed0f58d6d11e566845391051a09d8f15fa25f31a3432f" exitCode=0 Nov 03 03:08:45.865121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:45.864668 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" event={"ID":"75757f51-58f6-44f7-8a55-7ccd7b82ee28","Type":"ContainerDied","Data":"0b825a3565728dec730ed0f58d6d11e566845391051a09d8f15fa25f31a3432f"} Nov 03 03:08:46.871515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:46.871474 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" event={"ID":"75757f51-58f6-44f7-8a55-7ccd7b82ee28","Type":"ContainerStarted","Data":"6d54d630f71017304df1a7b5554c1a02c684b57bc6b74993b35708954d3ec333"} Nov 03 03:08:46.871949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:46.871682 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:08:47.137650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:47.137588 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" podStartSLOduration=2.616245363 podStartE2EDuration="8.137573868s" totalImagesPullingTime="5.05140749s" totalInitContainerRuntime="469.921015ms" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:47.087558039 +0000 UTC m=+418.240246461" watchObservedRunningTime="2026-11-03 03:08:47.137573868 +0000 UTC m=+418.290262289" Nov 03 03:08:51.841415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:51.841378 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z9kkw" Nov 03 03:08:57.879142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:08:57.879112 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-swhwj" Nov 03 03:09:00.936028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:00.935989 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" event={"ID":"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04","Type":"ContainerStarted","Data":"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c"} Nov 03 03:09:00.936453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:00.936138 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:09:01.102910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:01.102773 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" podStartSLOduration=2.010897875 podStartE2EDuration="19.102756986s" totalImagesPullingTime="17.091859111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:01.101554242 +0000 UTC m=+432.254242665" watchObservedRunningTime="2026-11-03 03:09:01.102756986 +0000 UTC m=+432.255445406" Nov 03 03:09:02.328463 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:09:02.328427 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:09:11.943235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:11.943206 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 03:09:13.791320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.791280 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-cbfxj"] Nov 03 03:09:13.795624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.795608 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:13.864137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.864097 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-cbfxj"] Nov 03 03:09:13.908779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.908751 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 03:09:13.908779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.908772 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 03:09:13.956216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.956181 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-bound-sa-token\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:13.956216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.956220 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w572w\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-kube-api-access-w572w\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:13.958526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:13.958503 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vmtvh\"" Nov 03 03:09:14.057140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.057061 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-bound-sa-token\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:14.057140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.057108 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w572w\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-kube-api-access-w572w\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:14.163052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.163019 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-bound-sa-token\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:14.166239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.166207 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w572w\" (UniqueName: \"kubernetes.io/projected/7eb36054-b42b-4736-911b-e65a5ddbc244-kube-api-access-w572w\") pod \"cert-manager-597b95cb84-cbfxj\" (UID: \"7eb36054-b42b-4736-911b-e65a5ddbc244\") " pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:14.240895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.240841 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:14.243241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.243223 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.301317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.301286 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8prwk\"" Nov 03 03:09:14.302518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.302499 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 03:09:14.333163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.333081 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:14.359962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.359932 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcl7q\" (UniqueName: \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.360161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.359983 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.418510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.418481 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-cbfxj" Nov 03 03:09:14.460682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.460651 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcl7q\" (UniqueName: \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.460868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.460696 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.461355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.461337 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.484909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.484695 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:14.581378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.581349 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcl7q\" (UniqueName: \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\") pod \"limitador-limitador-74dfcfb894-xzc8z\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:14.820281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.820248 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-cbfxj"] Nov 03 03:09:14.822992 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:14.822956 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eb36054_b42b_4736_911b_e65a5ddbc244.slice/crio-83e8d1c99217804beaeb45d0c967903cf1ae8fa5e4f05fe06c3641905c7cfe32 WatchSource:0}: Error finding container 83e8d1c99217804beaeb45d0c967903cf1ae8fa5e4f05fe06c3641905c7cfe32: Status 404 returned error can't find the container with id 83e8d1c99217804beaeb45d0c967903cf1ae8fa5e4f05fe06c3641905c7cfe32 Nov 03 03:09:14.855087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:14.855056 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:15.000109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.000075 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-cbfxj" event={"ID":"7eb36054-b42b-4736-911b-e65a5ddbc244","Type":"ContainerStarted","Data":"83e8d1c99217804beaeb45d0c967903cf1ae8fa5e4f05fe06c3641905c7cfe32"} Nov 03 03:09:15.085569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.085492 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll"] Nov 03 03:09:15.087980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.087964 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.178629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.178595 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vwlmw\"" Nov 03 03:09:15.195553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.195522 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll"] Nov 03 03:09:15.269459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269417 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269479 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269558 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269586 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pb2b\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-kube-api-access-2pb2b\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269613 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269705 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269812 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269834 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.269916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.269869 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.275866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.275808 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:15.279303 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:15.279255 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a5d848_a232_4c23_a67c_579766170d3e.slice/crio-ae94fa416fe09e0b2a50ed3698b8b6db5f4a640f057c814f136e6f23cd215a00 WatchSource:0}: Error finding container ae94fa416fe09e0b2a50ed3698b8b6db5f4a640f057c814f136e6f23cd215a00: Status 404 returned error can't find the container with id ae94fa416fe09e0b2a50ed3698b8b6db5f4a640f057c814f136e6f23cd215a00 Nov 03 03:09:15.371073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.370972 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371020 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371047 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371114 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371149 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371182 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371211 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb2b\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-kube-api-access-2pb2b\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371244 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371293 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.371663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.371486 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.372082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.372058 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.372205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.372089 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.372650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.372617 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.373138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.373114 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.374494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.374426 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.380395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.380365 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.455574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.455538 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.459667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.459637 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pb2b\" (UniqueName: \"kubernetes.io/projected/50cb5187-a18d-4f4b-87b9-e5a5c3c81873-kube-api-access-2pb2b\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-wnpll\" (UID: \"50cb5187-a18d-4f4b-87b9-e5a5c3c81873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:15.700785 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:15.700703 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:16.012225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:16.012141 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" event={"ID":"47a5d848-a232-4c23-a67c-579766170d3e","Type":"ContainerStarted","Data":"ae94fa416fe09e0b2a50ed3698b8b6db5f4a640f057c814f136e6f23cd215a00"} Nov 03 03:09:16.206449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:16.206410 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll"] Nov 03 03:09:16.209184 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:16.209143 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cb5187_a18d_4f4b_87b9_e5a5c3c81873.slice/crio-861ac03ab17900a073febc11ed889dfc4d4f5ef67de46e94bd8bbe314596a5c4 WatchSource:0}: Error finding container 861ac03ab17900a073febc11ed889dfc4d4f5ef67de46e94bd8bbe314596a5c4: Status 404 returned error can't find the container with id 861ac03ab17900a073febc11ed889dfc4d4f5ef67de46e94bd8bbe314596a5c4 Nov 03 03:09:16.211496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:16.211464 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:09:16.211569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:16.211531 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:09:16.211569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:16.211563 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:09:17.021754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:17.021719 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" event={"ID":"50cb5187-a18d-4f4b-87b9-e5a5c3c81873","Type":"ContainerStarted","Data":"f199df7f389295817c74cd41a3e2ae50cbe35487b9fc9e1e7d346a8a63c6476d"} Nov 03 03:09:17.022322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:17.021779 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" event={"ID":"50cb5187-a18d-4f4b-87b9-e5a5c3c81873","Type":"ContainerStarted","Data":"861ac03ab17900a073febc11ed889dfc4d4f5ef67de46e94bd8bbe314596a5c4"} Nov 03 03:09:17.701266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:17.701234 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:17.706554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:17.706493 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:17.861926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:17.861873 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" podStartSLOduration=2.861840276 podStartE2EDuration="2.861840276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:17.208245294 +0000 UTC m=+448.360933716" watchObservedRunningTime="2026-11-03 03:09:17.861840276 +0000 UTC m=+449.014528698" Nov 03 03:09:18.028947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:18.028913 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" event={"ID":"47a5d848-a232-4c23-a67c-579766170d3e","Type":"ContainerStarted","Data":"24de94708827e83e7c146caa6358de4df1e568e1755214a1c2a2b5abbcc1481c"} Nov 03 03:09:18.029375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:18.029181 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:18.029375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:18.029218 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:18.030096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:18.030069 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-wnpll" Nov 03 03:09:18.174901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:18.174766 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" podStartSLOduration=1.976631468 podStartE2EDuration="4.174746517s" totalImagesPullingTime="2.198115049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:18.173498606 +0000 UTC m=+449.326187028" watchObservedRunningTime="2026-11-03 03:09:18.174746517 +0000 UTC m=+449.327434940" Nov 03 03:09:19.475335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.475300 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:09:19.478242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.478218 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:09:19.512207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.512168 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd78\" (UniqueName: \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\") pod \"authorino-56c6c9ddd5-8rzzn\" (UID: \"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598\") " pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:09:19.545249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.545212 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84rtq\"" Nov 03 03:09:19.596419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.596392 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:09:19.613205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.613171 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd78\" (UniqueName: \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\") pod \"authorino-56c6c9ddd5-8rzzn\" (UID: \"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598\") " pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:09:19.676409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.676373 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd78\" (UniqueName: \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\") pod \"authorino-56c6c9ddd5-8rzzn\" (UID: \"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598\") " pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:09:19.789554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:19.789522 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:09:20.042901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:20.042795 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-cbfxj" event={"ID":"7eb36054-b42b-4736-911b-e65a5ddbc244","Type":"ContainerStarted","Data":"2f9189af256d22e1bed09dea2effc3962faaae6a38ad2d67cec774cd55b0b639"} Nov 03 03:09:20.164864 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:20.164791 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-cbfxj" podStartSLOduration=2.402624987 podStartE2EDuration="7.164776067s" totalImagesPullingTime="4.76215108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:20.163322682 +0000 UTC m=+451.316011104" watchObservedRunningTime="2026-11-03 03:09:20.164776067 +0000 UTC m=+451.317464488" Nov 03 03:09:20.171584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:20.171550 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:09:20.175988 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:20.175956 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ae5fc8_7ce3_4c84_b35b_bf324b9be598.slice/crio-90aa033b4ed3d0be552b0ef826bf0724eb1d5dd97fc07e33ba342aaca284a0f7 WatchSource:0}: Error finding container 90aa033b4ed3d0be552b0ef826bf0724eb1d5dd97fc07e33ba342aaca284a0f7: Status 404 returned error can't find the container with id 90aa033b4ed3d0be552b0ef826bf0724eb1d5dd97fc07e33ba342aaca284a0f7 Nov 03 03:09:21.048471 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:21.048437 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" event={"ID":"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598","Type":"ContainerStarted","Data":"90aa033b4ed3d0be552b0ef826bf0724eb1d5dd97fc07e33ba342aaca284a0f7"} Nov 03 03:09:23.059884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:23.059830 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" event={"ID":"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598","Type":"ContainerStarted","Data":"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b"} Nov 03 03:09:23.175171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:23.175103 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" podStartSLOduration=1.718185134 podStartE2EDuration="4.175085454s" totalImagesPullingTime="2.45690032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:23.174331562 +0000 UTC m=+454.327019984" watchObservedRunningTime="2026-11-03 03:09:23.175085454 +0000 UTC m=+454.327773875" Nov 03 03:09:29.035524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:29.035493 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:32.327789 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:09:32.327748 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:09:33.680386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:33.680344 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:33.680886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:33.680584 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" podUID="47a5d848-a232-4c23-a67c-579766170d3e" containerName="limitador" containerID="cri-o://24de94708827e83e7c146caa6358de4df1e568e1755214a1c2a2b5abbcc1481c" gracePeriod=30 Nov 03 03:09:34.113566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.113524 5275 generic.go:356] "Generic (PLEG): container finished" podID="47a5d848-a232-4c23-a67c-579766170d3e" containerID="24de94708827e83e7c146caa6358de4df1e568e1755214a1c2a2b5abbcc1481c" exitCode=0 Nov 03 03:09:34.113757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.113595 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" event={"ID":"47a5d848-a232-4c23-a67c-579766170d3e","Type":"ContainerDied","Data":"24de94708827e83e7c146caa6358de4df1e568e1755214a1c2a2b5abbcc1481c"} Nov 03 03:09:34.280212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.280182 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:34.332739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.332705 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\" (UniqueName: \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\") pod \"47a5d848-a232-4c23-a67c-579766170d3e\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " Nov 03 03:09:34.332959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.332761 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\" (UniqueName: \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\") pod \"47a5d848-a232-4c23-a67c-579766170d3e\" (UID: \"47a5d848-a232-4c23-a67c-579766170d3e\") " Nov 03 03:09:34.333123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.333091 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file" pod "47a5d848-a232-4c23-a67c-579766170d3e" (UID: "47a5d848-a232-4c23-a67c-579766170d3e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:34.335262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.335237 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q" pod "47a5d848-a232-4c23-a67c-579766170d3e" (UID: "47a5d848-a232-4c23-a67c-579766170d3e"). InnerVolumeSpecName "kube-api-access-mcl7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:34.433559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.433522 5275 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47a5d848-a232-4c23-a67c-579766170d3e-config-file\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:09:34.433559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:34.433553 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcl7q\" (UniqueName: \"kubernetes.io/projected/47a5d848-a232-4c23-a67c-579766170d3e-kube-api-access-mcl7q\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:09:35.120232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:35.120203 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" Nov 03 03:09:35.120232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:35.120223 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xzc8z" event={"ID":"47a5d848-a232-4c23-a67c-579766170d3e","Type":"ContainerDied","Data":"ae94fa416fe09e0b2a50ed3698b8b6db5f4a640f057c814f136e6f23cd215a00"} Nov 03 03:09:35.120786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:35.120292 5275 scope.go:122] "RemoveContainer" containerID="24de94708827e83e7c146caa6358de4df1e568e1755214a1c2a2b5abbcc1481c" Nov 03 03:09:35.285662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:35.285633 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:35.335065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:35.335035 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xzc8z"] Nov 03 03:09:36.986132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:36.986089 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a5d848-a232-4c23-a67c-579766170d3e" path="/var/lib/kubelet/pods/47a5d848-a232-4c23-a67c-579766170d3e/volumes" Nov 03 03:09:42.708499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.708466 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z"] Nov 03 03:09:42.709024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.708793 5275 cpu_manager.go:414] "containerMap: removing container" podUID="47a5d848-a232-4c23-a67c-579766170d3e" containerName="limitador" Nov 03 03:09:42.709024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.708803 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a5d848-a232-4c23-a67c-579766170d3e" containerName="limitador" Nov 03 03:09:42.709024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.708905 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a5d848-a232-4c23-a67c-579766170d3e" containerName="limitador" Nov 03 03:09:42.711409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.711388 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.763944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.763907 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 03:09:42.763944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.763926 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 03:09:42.806216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.806177 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvf8w\" (UniqueName: \"kubernetes.io/projected/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-kube-api-access-kvf8w\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.806392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.806215 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.806392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.806341 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.806392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.806363 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907221 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907267 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907329 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvf8w\" (UniqueName: \"kubernetes.io/projected/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-kube-api-access-kvf8w\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907354 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907629 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.907689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.907662 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.919126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.913758 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 03:09:42.919126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.913837 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9k8c2\"" Nov 03 03:09:42.919126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.919085 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z"] Nov 03 03:09:42.920240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.920219 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:42.969202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:42.969124 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvf8w\" (UniqueName: \"kubernetes.io/projected/c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec-kube-api-access-kvf8w\") pod \"kserve-module-controller-manager-7566cf5fc8-l6m8z\" (UID: \"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:43.022326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:43.022295 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:43.446552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:43.446526 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z"] Nov 03 03:09:43.448473 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:43.448445 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eafc7c_7fa8_4b09_8cb5_0852f9b7d4ec.slice/crio-cbb40cdd0cfd561f3990f9f2c5d558c1cf510562b8dca1bc86b6d837da681eb0 WatchSource:0}: Error finding container cbb40cdd0cfd561f3990f9f2c5d558c1cf510562b8dca1bc86b6d837da681eb0: Status 404 returned error can't find the container with id cbb40cdd0cfd561f3990f9f2c5d558c1cf510562b8dca1bc86b6d837da681eb0 Nov 03 03:09:44.164016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:44.163980 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" event={"ID":"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec","Type":"ContainerStarted","Data":"cbb40cdd0cfd561f3990f9f2c5d558c1cf510562b8dca1bc86b6d837da681eb0"} Nov 03 03:09:47.181911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:47.181869 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" event={"ID":"c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec","Type":"ContainerStarted","Data":"44eb77422360fa0c7b4bea0f506189338b56fb51a2fa5bf5abc891846d17216a"} Nov 03 03:09:47.352456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:47.352393 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" podStartSLOduration=2.485132409 podStartE2EDuration="5.352378573s" totalImagesPullingTime="2.867246164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:47.351609526 +0000 UTC m=+478.504297984" watchObservedRunningTime="2026-11-03 03:09:47.352378573 +0000 UTC m=+478.505066994" Nov 03 03:09:48.186250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:48.186220 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:09:53.390877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.390809 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js"] Nov 03 03:09:53.395164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.395129 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.503446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503409 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 03:09:53.503446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503409 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:09:53.503677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503472 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xf54k\"" Nov 03 03:09:53.503677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503499 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 03:09:53.503677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503515 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 03:09:53.503677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503528 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 03:09:53.503911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503763 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nch\" (UniqueName: \"kubernetes.io/projected/73bccfc4-f16b-4569-a216-971d144c0346-kube-api-access-t9nch\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.503911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503839 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-cert\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.504013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.503938 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-metrics-certs\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.504101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.504080 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73bccfc4-f16b-4569-a216-971d144c0346-manager-config\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.505288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.505262 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js"] Nov 03 03:09:53.605559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.605525 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-cert\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.606273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.606248 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-metrics-certs\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.606703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.606688 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73bccfc4-f16b-4569-a216-971d144c0346-manager-config\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.607504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.607485 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nch\" (UniqueName: \"kubernetes.io/projected/73bccfc4-f16b-4569-a216-971d144c0346-kube-api-access-t9nch\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.608094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.607414 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73bccfc4-f16b-4569-a216-971d144c0346-manager-config\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.616029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.616002 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-cert\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.616540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.616520 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73bccfc4-f16b-4569-a216-971d144c0346-metrics-certs\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.675514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.675438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nch\" (UniqueName: \"kubernetes.io/projected/73bccfc4-f16b-4569-a216-971d144c0346-kube-api-access-t9nch\") pod \"jobset-controller-manager-545b47c484-pl2js\" (UID: \"73bccfc4-f16b-4569-a216-971d144c0346\") " pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:53.707275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:53.707230 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:54.106768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:54.106737 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js"] Nov 03 03:09:54.109175 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:09:54.109150 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bccfc4_f16b_4569_a216_971d144c0346.slice/crio-ce01424ae711eabfbb026dfb39d8a64bf681ea84d5152c69964b3ed30ec24d11 WatchSource:0}: Error finding container ce01424ae711eabfbb026dfb39d8a64bf681ea84d5152c69964b3ed30ec24d11: Status 404 returned error can't find the container with id ce01424ae711eabfbb026dfb39d8a64bf681ea84d5152c69964b3ed30ec24d11 Nov 03 03:09:54.215710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:54.215674 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" event={"ID":"73bccfc4-f16b-4569-a216-971d144c0346","Type":"ContainerStarted","Data":"ce01424ae711eabfbb026dfb39d8a64bf681ea84d5152c69964b3ed30ec24d11"} Nov 03 03:09:58.237702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:58.237619 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" event={"ID":"73bccfc4-f16b-4569-a216-971d144c0346","Type":"ContainerStarted","Data":"218edc21cb387d021cd56a0b4115492a4fe3d0d75ed367655e3cf127a8fcaa68"} Nov 03 03:09:58.238111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:58.237737 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:09:58.354619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:09:58.354561 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" podStartSLOduration=1.6102089419999999 podStartE2EDuration="5.354545964s" totalImagesPullingTime="3.744337022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:58.353992108 +0000 UTC m=+489.506680530" watchObservedRunningTime="2026-11-03 03:09:58.354545964 +0000 UTC m=+489.507234424" Nov 03 03:10:02.328991 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:10:02.328954 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:10:09.192340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:09.192309 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l6m8z" Nov 03 03:10:09.247556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:09.247520 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-545b47c484-pl2js" Nov 03 03:10:32.328614 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:10:32.328509 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:10:50.990580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:50.990546 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lskgg"] Nov 03 03:10:50.997756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:50.997734 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.045576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.045543 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lskgg"] Nov 03 03:10:51.093226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.093199 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2wjwq\"" Nov 03 03:10:51.113163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.113130 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 03:10:51.130565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.130535 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkdp\" (UniqueName: \"kubernetes.io/projected/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-kube-api-access-lrkdp\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.130724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.130602 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-tls-certs\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.231662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.231624 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkdp\" (UniqueName: \"kubernetes.io/projected/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-kube-api-access-lrkdp\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.231869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.231704 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-tls-certs\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.234285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.234262 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-tls-certs\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.293437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.293409 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkdp\" (UniqueName: \"kubernetes.io/projected/f0d5b72d-3781-4776-8dfe-4b2728d5d42e-kube-api-access-lrkdp\") pod \"model-serving-api-75bd89fdfc-lskgg\" (UID: \"f0d5b72d-3781-4776-8dfe-4b2728d5d42e\") " pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.310267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.310239 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:51.707378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:51.707356 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lskgg"] Nov 03 03:10:51.709582 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:10:51.709552 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d5b72d_3781_4776_8dfe_4b2728d5d42e.slice/crio-229ed8051c284c52f25bcc489a9edd584a3cea1074039bcabd671c915bd943e0 WatchSource:0}: Error finding container 229ed8051c284c52f25bcc489a9edd584a3cea1074039bcabd671c915bd943e0: Status 404 returned error can't find the container with id 229ed8051c284c52f25bcc489a9edd584a3cea1074039bcabd671c915bd943e0 Nov 03 03:10:52.512290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:52.512211 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" event={"ID":"f0d5b72d-3781-4776-8dfe-4b2728d5d42e","Type":"ContainerStarted","Data":"229ed8051c284c52f25bcc489a9edd584a3cea1074039bcabd671c915bd943e0"} Nov 03 03:10:55.531077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:55.531017 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" event={"ID":"f0d5b72d-3781-4776-8dfe-4b2728d5d42e","Type":"ContainerStarted","Data":"3b0d83815b208e82c607c3f551bf3e2e550a9cfdecf2de0af90fdfa92d6647b9"} Nov 03 03:10:55.531476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:55.531143 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:10:55.658874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:55.649738 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" podStartSLOduration=2.724025799 podStartE2EDuration="5.649716181s" totalImagesPullingTime="2.925690382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:55.647892044 +0000 UTC m=+546.800580465" watchObservedRunningTime="2026-11-03 03:10:55.649716181 +0000 UTC m=+546.802404603" Nov 03 03:10:56.397539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.397508 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:56.401084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.401067 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:56.458276 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.458245 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:56.504916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.504881 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8d7\" (UniqueName: \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\") pod \"authorino-6989f87447-jfdv8\" (UID: \"979e32d7-346e-4804-a312-341519fff9b3\") " pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:56.605807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.605766 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8d7\" (UniqueName: \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\") pod \"authorino-6989f87447-jfdv8\" (UID: \"979e32d7-346e-4804-a312-341519fff9b3\") " pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:56.670318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.670246 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8d7\" (UniqueName: \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\") pod \"authorino-6989f87447-jfdv8\" (UID: \"979e32d7-346e-4804-a312-341519fff9b3\") " pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:56.713119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.713087 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:56.784249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.783970 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:56.818552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.818517 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:10:56.822826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.822804 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:56.878169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.878134 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 03:10:56.879674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:56.879642 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:10:57.010302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.010202 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mq99\" (UniqueName: \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.010302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.010268 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.087182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.087150 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:57.090375 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:10:57.090347 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod979e32d7_346e_4804_a312_341519fff9b3.slice/crio-1c1614a0bd1065b82ce1e3252f86fc10f85e3bf38c9cc9656c4be9fed4175cd7 WatchSource:0}: Error finding container 1c1614a0bd1065b82ce1e3252f86fc10f85e3bf38c9cc9656c4be9fed4175cd7: Status 404 returned error can't find the container with id 1c1614a0bd1065b82ce1e3252f86fc10f85e3bf38c9cc9656c4be9fed4175cd7 Nov 03 03:10:57.111469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.111431 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.111635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.111599 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mq99\" (UniqueName: \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.114098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.114076 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.173339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.173305 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mq99\" (UniqueName: \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\") pod \"authorino-64797fcd5-mmjpq\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.435586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.435544 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:10:57.542913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.542876 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6989f87447-jfdv8" event={"ID":"979e32d7-346e-4804-a312-341519fff9b3","Type":"ContainerStarted","Data":"1c1614a0bd1065b82ce1e3252f86fc10f85e3bf38c9cc9656c4be9fed4175cd7"} Nov 03 03:10:57.561077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.561032 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:10:57.561246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.561128 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:10:57.812956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:57.812685 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:10:57.815671 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:10:57.815643 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda7e79c5_6cbc_49d7_b4c4_b531c930f98d.slice/crio-efcbaf4b6eafe5bd5a31679ebe962fa396ce7d222c57f70ef871679ce173b79c WatchSource:0}: Error finding container efcbaf4b6eafe5bd5a31679ebe962fa396ce7d222c57f70ef871679ce173b79c: Status 404 returned error can't find the container with id efcbaf4b6eafe5bd5a31679ebe962fa396ce7d222c57f70ef871679ce173b79c Nov 03 03:10:58.549817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.549776 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6989f87447-jfdv8" event={"ID":"979e32d7-346e-4804-a312-341519fff9b3","Type":"ContainerStarted","Data":"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab"} Nov 03 03:10:58.550016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.549878 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6989f87447-jfdv8" podUID="979e32d7-346e-4804-a312-341519fff9b3" containerName="authorino" containerID="cri-o://aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab" gracePeriod=30 Nov 03 03:10:58.551317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.551291 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64797fcd5-mmjpq" event={"ID":"da7e79c5-6cbc-49d7-b4c4-b531c930f98d","Type":"ContainerStarted","Data":"efcbaf4b6eafe5bd5a31679ebe962fa396ce7d222c57f70ef871679ce173b79c"} Nov 03 03:10:58.665357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.665307 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6989f87447-jfdv8" podStartSLOduration=1.489665929 podStartE2EDuration="2.665292052s" totalImagesPullingTime="1.175626123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:58.664304885 +0000 UTC m=+549.816993307" watchObservedRunningTime="2026-11-03 03:10:58.665292052 +0000 UTC m=+549.817980473" Nov 03 03:10:58.803283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.803221 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:58.929351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.929317 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\" (UniqueName: \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\") pod \"979e32d7-346e-4804-a312-341519fff9b3\" (UID: \"979e32d7-346e-4804-a312-341519fff9b3\") " Nov 03 03:10:58.931680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:58.931649 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7" pod "979e32d7-346e-4804-a312-341519fff9b3" (UID: "979e32d7-346e-4804-a312-341519fff9b3"). InnerVolumeSpecName "kube-api-access-2r8d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:59.030928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.030887 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8d7\" (UniqueName: \"kubernetes.io/projected/979e32d7-346e-4804-a312-341519fff9b3-kube-api-access-2r8d7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:10:59.557357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.557323 5275 generic.go:356] "Generic (PLEG): container finished" podID="979e32d7-346e-4804-a312-341519fff9b3" containerID="aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab" exitCode=0 Nov 03 03:10:59.559334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.557372 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6989f87447-jfdv8" Nov 03 03:10:59.559334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.557405 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6989f87447-jfdv8" event={"ID":"979e32d7-346e-4804-a312-341519fff9b3","Type":"ContainerDied","Data":"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab"} Nov 03 03:10:59.559334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.557441 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6989f87447-jfdv8" event={"ID":"979e32d7-346e-4804-a312-341519fff9b3","Type":"ContainerDied","Data":"1c1614a0bd1065b82ce1e3252f86fc10f85e3bf38c9cc9656c4be9fed4175cd7"} Nov 03 03:10:59.559334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.557462 5275 scope.go:122] "RemoveContainer" containerID="aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab" Nov 03 03:10:59.559334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.559171 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64797fcd5-mmjpq" event={"ID":"da7e79c5-6cbc-49d7-b4c4-b531c930f98d","Type":"ContainerStarted","Data":"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b"} Nov 03 03:10:59.567645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.567624 5275 scope.go:122] "RemoveContainer" containerID="aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab" Nov 03 03:10:59.567985 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:10:59.567965 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab\": container with ID starting with aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab not found: ID does not exist" containerID="aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab" Nov 03 03:10:59.568085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.567993 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab"} err="failed to get container status \"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab\": rpc error: code = NotFound desc = could not find container \"aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab\": container with ID starting with aa50cfea2fa8da8bf9bdc88654cc4961c30d4f7dafef8247d49311eb22e13bab not found: ID does not exist" Nov 03 03:10:59.723828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.723715 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:59.725809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.725781 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6989f87447-jfdv8"] Nov 03 03:10:59.839223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.839107 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-64797fcd5-mmjpq" podStartSLOduration=2.517311782 podStartE2EDuration="3.839086885s" totalImagesPullingTime="1.321775103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:59.838019587 +0000 UTC m=+550.990708010" watchObservedRunningTime="2026-11-03 03:10:59.839086885 +0000 UTC m=+550.991775308" Nov 03 03:10:59.877400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.875988 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:10:59.877400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:10:59.876207 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" podUID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" containerName="authorino" containerID="cri-o://9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b" gracePeriod=30 Nov 03 03:11:00.134411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.134380 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:11:00.141461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.141437 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\" (UniqueName: \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\") pod \"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598\" (UID: \"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598\") " Nov 03 03:11:00.143740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.143704 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78" pod "e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" (UID: "e4ae5fc8-7ce3-4c84-b35b-bf324b9be598"). InnerVolumeSpecName "kube-api-access-qhd78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:00.242770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.242722 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhd78\" (UniqueName: \"kubernetes.io/projected/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598-kube-api-access-qhd78\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:00.566928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.566889 5275 generic.go:356] "Generic (PLEG): container finished" podID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" containerID="9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b" exitCode=0 Nov 03 03:11:00.566928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.566927 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" Nov 03 03:11:00.567150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.566962 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" event={"ID":"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598","Type":"ContainerDied","Data":"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b"} Nov 03 03:11:00.567150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.567015 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8rzzn" event={"ID":"e4ae5fc8-7ce3-4c84-b35b-bf324b9be598","Type":"ContainerDied","Data":"90aa033b4ed3d0be552b0ef826bf0724eb1d5dd97fc07e33ba342aaca284a0f7"} Nov 03 03:11:00.567150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.567041 5275 scope.go:122] "RemoveContainer" containerID="9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b" Nov 03 03:11:00.568814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.568755 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerID="85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb" exitCode=0 Nov 03 03:11:00.568984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.568941 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerDied","Data":"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb"} Nov 03 03:11:00.569356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.569336 5275 scope.go:122] "RemoveContainer" containerID="85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb" Nov 03 03:11:00.570469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.570440 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:11:00.570573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.570515 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:11:00.570573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.570552 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:11:00.576992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.576977 5275 scope.go:122] "RemoveContainer" containerID="9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b" Nov 03 03:11:00.577271 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:11:00.577255 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b\": container with ID starting with 9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b not found: ID does not exist" containerID="9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b" Nov 03 03:11:00.577331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.577283 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b"} err="failed to get container status \"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b\": rpc error: code = NotFound desc = could not find container \"9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b\": container with ID starting with 9e16ce47e156ff4f4d55401c117ff0d1ac36baa6433fc1889deed01696ebf88b not found: ID does not exist" Nov 03 03:11:00.732650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.732617 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:11:00.736768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.736743 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8rzzn"] Nov 03 03:11:00.994041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.993964 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979e32d7-346e-4804-a312-341519fff9b3" path="/var/lib/kubelet/pods/979e32d7-346e-4804-a312-341519fff9b3/volumes" Nov 03 03:11:00.994309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:00.994297 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" path="/var/lib/kubelet/pods/e4ae5fc8-7ce3-4c84-b35b-bf324b9be598/volumes" Nov 03 03:11:01.581478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:01.581440 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerStarted","Data":"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952"} Nov 03 03:11:02.328486 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:11:02.328436 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:11:02.477419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:02.477384 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:11:02.477675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:02.477646 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 03:11:02.477748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:02.477697 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 03:11:03.477054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:03.477015 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 03:11:03.477574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:03.477096 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 03:11:04.477927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:04.477892 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 03:11:04.478340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:04.477945 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 03:11:05.477469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:05.477432 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 03:11:05.477654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:05.477487 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 03:11:06.477770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:06.477734 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 03:11:06.478284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:06.477814 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 03:11:06.540550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:06.540523 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lskgg" Nov 03 03:11:07.481053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:07.481024 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:11:07.481450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:07.481300 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:11:07.482079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:07.482062 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:11:32.328342 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:11:32.328303 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:11:49.030503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:49.030479 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:11:49.038684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:11:49.038662 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:12:02.328875 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:12:02.328825 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:12:32.328602 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:12:32.328513 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:12:35.762211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762175 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:12:35.762601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762536 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" containerName="authorino" Nov 03 03:12:35.762601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762547 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" containerName="authorino" Nov 03 03:12:35.762601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762562 5275 cpu_manager.go:414] "containerMap: removing container" podUID="979e32d7-346e-4804-a312-341519fff9b3" containerName="authorino" Nov 03 03:12:35.762601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762567 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979e32d7-346e-4804-a312-341519fff9b3" containerName="authorino" Nov 03 03:12:35.762727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762637 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae5fc8-7ce3-4c84-b35b-bf324b9be598" containerName="authorino" Nov 03 03:12:35.762727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.762647 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="979e32d7-346e-4804-a312-341519fff9b3" containerName="authorino" Nov 03 03:12:35.765515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.765499 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.822612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.822563 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlkfc\" (UniqueName: \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.823272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.823059 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.867385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.867345 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 03:12:35.867385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.867343 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-skt72\"" Nov 03 03:12:35.867636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.867420 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 03:12:35.869307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.869282 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:12:35.936731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.936693 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlkfc\" (UniqueName: \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.937422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.937395 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.941498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.941471 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:35.999057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:35.999024 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlkfc\" (UniqueName: \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\") pod \"ai-gateway-controller-5d8dddb676-k7phw\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:36.086054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:36.086015 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:36.460891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:36.460865 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:12:36.463432 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:12:36.463404 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8a9d74_842b_4cfa_b49d_07449850cfa6.slice/crio-4dc7e06582985d7c49e05a585155fc06985f5a576384ca42e3a7a7eaca5c43d3 WatchSource:0}: Error finding container 4dc7e06582985d7c49e05a585155fc06985f5a576384ca42e3a7a7eaca5c43d3: Status 404 returned error can't find the container with id 4dc7e06582985d7c49e05a585155fc06985f5a576384ca42e3a7a7eaca5c43d3 Nov 03 03:12:37.041005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:37.040972 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" event={"ID":"4b8a9d74-842b-4cfa-b49d-07449850cfa6","Type":"ContainerStarted","Data":"4dc7e06582985d7c49e05a585155fc06985f5a576384ca42e3a7a7eaca5c43d3"} Nov 03 03:12:40.059255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:40.059213 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" event={"ID":"4b8a9d74-842b-4cfa-b49d-07449850cfa6","Type":"ContainerStarted","Data":"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12"} Nov 03 03:12:40.059784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:40.059357 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:40.177182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:40.177122 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" podStartSLOduration=2.208042738 podStartE2EDuration="5.17710544s" totalImagesPullingTime="2.969062702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:40.176757193 +0000 UTC m=+651.329445614" watchObservedRunningTime="2026-11-03 03:12:40.17710544 +0000 UTC m=+651.329793862" Nov 03 03:12:43.960099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:43.960055 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 03:12:43.963950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:43.963924 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.009390 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.009352 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgf8j\" (UniqueName: \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.009607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.009513 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.016951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.016923 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2h5gl\"" Nov 03 03:12:44.066258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.066225 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 03:12:44.067914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.067875 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 03:12:44.110335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.110307 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.110509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.110356 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgf8j\" (UniqueName: \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.111298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.111275 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.172152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.172119 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgf8j\" (UniqueName: \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\") pod \"payload-processing-55fbbbf9dc-znlwv\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.277094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.277065 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:44.649358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:44.649331 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 03:12:44.651958 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:12:44.651933 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ec2be7_fff9_4367_b67e_83e32e0f0586.slice/crio-289cb0614abc436a42d67abc44b6a48d74f4cf1dacf2023e1cacd949a40c2508 WatchSource:0}: Error finding container 289cb0614abc436a42d67abc44b6a48d74f4cf1dacf2023e1cacd949a40c2508: Status 404 returned error can't find the container with id 289cb0614abc436a42d67abc44b6a48d74f4cf1dacf2023e1cacd949a40c2508 Nov 03 03:12:45.084816 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:45.084778 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" event={"ID":"59ec2be7-fff9-4367-b67e-83e32e0f0586","Type":"ContainerStarted","Data":"289cb0614abc436a42d67abc44b6a48d74f4cf1dacf2023e1cacd949a40c2508"} Nov 03 03:12:48.104857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:48.104802 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" event={"ID":"59ec2be7-fff9-4367-b67e-83e32e0f0586","Type":"ContainerStarted","Data":"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983"} Nov 03 03:12:48.105255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:48.104872 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:12:48.222493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:48.222433 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" podStartSLOduration=2.549437906 podStartE2EDuration="5.222411801s" totalImagesPullingTime="2.672973895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:48.221142502 +0000 UTC m=+659.373830925" watchObservedRunningTime="2026-11-03 03:12:48.222411801 +0000 UTC m=+659.375100220" Nov 03 03:12:51.066533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:51.066504 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:12:59.113371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:12:59.113338 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 03:13:02.331859 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:13:02.331799 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:13:06.565455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.565421 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:13:06.565887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.565637 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" podUID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" containerName="manager" containerID="cri-o://233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12" gracePeriod=30 Nov 03 03:13:06.812043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.812017 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:13:06.914973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.914831 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\" (UniqueName: \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\") pod \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " Nov 03 03:13:06.914973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.914921 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\") pod \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\" (UID: \"4b8a9d74-842b-4cfa-b49d-07449850cfa6\") " Nov 03 03:13:06.917278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.917250 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc" pod "4b8a9d74-842b-4cfa-b49d-07449850cfa6" (UID: "4b8a9d74-842b-4cfa-b49d-07449850cfa6"). InnerVolumeSpecName "kube-api-access-mlkfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:06.917366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:06.917283 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs" pod "4b8a9d74-842b-4cfa-b49d-07449850cfa6" (UID: "4b8a9d74-842b-4cfa-b49d-07449850cfa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:07.016338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.016301 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlkfc\" (UniqueName: \"kubernetes.io/projected/4b8a9d74-842b-4cfa-b49d-07449850cfa6-kube-api-access-mlkfc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:13:07.016338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.016332 5275 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8a9d74-842b-4cfa-b49d-07449850cfa6-metrics-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:13:07.202731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.202626 5275 generic.go:356] "Generic (PLEG): container finished" podID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" containerID="233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12" exitCode=0 Nov 03 03:13:07.202731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.202686 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" Nov 03 03:13:07.203000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.202726 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" event={"ID":"4b8a9d74-842b-4cfa-b49d-07449850cfa6","Type":"ContainerDied","Data":"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12"} Nov 03 03:13:07.203000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.202804 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-k7phw" event={"ID":"4b8a9d74-842b-4cfa-b49d-07449850cfa6","Type":"ContainerDied","Data":"4dc7e06582985d7c49e05a585155fc06985f5a576384ca42e3a7a7eaca5c43d3"} Nov 03 03:13:07.203000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.202827 5275 scope.go:122] "RemoveContainer" containerID="233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12" Nov 03 03:13:07.211426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.211406 5275 scope.go:122] "RemoveContainer" containerID="233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12" Nov 03 03:13:07.211721 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:13:07.211702 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12\": container with ID starting with 233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12 not found: ID does not exist" containerID="233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12" Nov 03 03:13:07.211763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.211731 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12"} err="failed to get container status \"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12\": rpc error: code = NotFound desc = could not find container \"233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12\": container with ID starting with 233d85eb9b9fc29d2ebdf7684d61ee96e1a50077344ff647420b38fdc26a7f12 not found: ID does not exist" Nov 03 03:13:07.367419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.367386 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:13:07.416810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:07.416778 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-k7phw"] Nov 03 03:13:08.982266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:08.982230 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" path="/var/lib/kubelet/pods/4b8a9d74-842b-4cfa-b49d-07449850cfa6/volumes" Nov 03 03:13:18.118705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.118676 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf"] Nov 03 03:13:18.119185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.119029 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" containerName="manager" Nov 03 03:13:18.119185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.119040 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" containerName="manager" Nov 03 03:13:18.119185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.119112 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8a9d74-842b-4cfa-b49d-07449850cfa6" containerName="manager" Nov 03 03:13:18.127864 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.127824 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.182298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.182269 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf"] Nov 03 03:13:18.229489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.229452 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 03:13:18.229489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.229486 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b5ckh\"" Nov 03 03:13:18.229489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.229463 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 03:13:18.229737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.229530 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 03:13:18.315397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315359 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.315397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315398 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc2t\" (UniqueName: \"kubernetes.io/projected/65265c30-274d-45b6-9222-05907af3e477-kube-api-access-nmc2t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.315632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315565 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.315632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315603 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.315700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315655 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.315700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.315693 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65265c30-274d-45b6-9222-05907af3e477-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.416881 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.416952 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65265c30-274d-45b6-9222-05907af3e477-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417002 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417037 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc2t\" (UniqueName: \"kubernetes.io/projected/65265c30-274d-45b6-9222-05907af3e477-kube-api-access-nmc2t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417119 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417161 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417584 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.418319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.417833 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.422030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.421644 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65265c30-274d-45b6-9222-05907af3e477-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.422841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.422805 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.424344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.424322 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65265c30-274d-45b6-9222-05907af3e477-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.484838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.484810 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc2t\" (UniqueName: \"kubernetes.io/projected/65265c30-274d-45b6-9222-05907af3e477-kube-api-access-nmc2t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf\" (UID: \"65265c30-274d-45b6-9222-05907af3e477\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:18.742082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:18.741998 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:19.124354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:19.124331 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf"] Nov 03 03:13:19.126753 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:13:19.126732 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65265c30_274d_45b6_9222_05907af3e477.slice/crio-44d989d5a603f9005fbd31efbdef70d89791e345e41c706c0000e51fac284755 WatchSource:0}: Error finding container 44d989d5a603f9005fbd31efbdef70d89791e345e41c706c0000e51fac284755: Status 404 returned error can't find the container with id 44d989d5a603f9005fbd31efbdef70d89791e345e41c706c0000e51fac284755 Nov 03 03:13:19.128073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:19.128058 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:13:19.265519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:19.265486 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" event={"ID":"65265c30-274d-45b6-9222-05907af3e477","Type":"ContainerStarted","Data":"44d989d5a603f9005fbd31efbdef70d89791e345e41c706c0000e51fac284755"} Nov 03 03:13:22.281358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:22.280888 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" event={"ID":"65265c30-274d-45b6-9222-05907af3e477","Type":"ContainerStarted","Data":"965341362d1cdfd575e845a27dadbecdda9daf43f77ef35442e45832d7546227"} Nov 03 03:13:22.402652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:22.402586 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" podStartSLOduration=2.201714779 podStartE2EDuration="4.402567154s" totalImagesPullingTime="2.200852375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:22.400254044 +0000 UTC m=+693.552942467" watchObservedRunningTime="2026-11-03 03:13:22.402567154 +0000 UTC m=+693.555255575" Nov 03 03:13:22.742601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:22.742500 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:28.742925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:28.742885 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:28.755669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:28.755636 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:29.326436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:29.326407 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf" Nov 03 03:13:32.329871 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:13:32.329803 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:13:34.848887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.848836 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg"] Nov 03 03:13:34.852328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.852305 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.954119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.954088 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 03:13:34.956877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.956834 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg"] Nov 03 03:13:34.960430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960400 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.960558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960446 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2s8d\" (UniqueName: \"kubernetes.io/projected/8531b165-d56f-4bdf-893a-b2395e9dd85e-kube-api-access-w2s8d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.960558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960549 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.960631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960583 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.960631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960610 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:34.960749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:34.960728 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8531b165-d56f-4bdf-893a-b2395e9dd85e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.061761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061728 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.061969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061772 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.061969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061810 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8531b165-d56f-4bdf-893a-b2395e9dd85e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.061969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061870 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.061969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061922 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2s8d\" (UniqueName: \"kubernetes.io/projected/8531b165-d56f-4bdf-893a-b2395e9dd85e-kube-api-access-w2s8d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.062189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.061982 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.062304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.062279 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.062403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.062280 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.062403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.062357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.064293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.064271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8531b165-d56f-4bdf-893a-b2395e9dd85e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.064579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.064557 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8531b165-d56f-4bdf-893a-b2395e9dd85e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.127987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.127911 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2s8d\" (UniqueName: \"kubernetes.io/projected/8531b165-d56f-4bdf-893a-b2395e9dd85e-kube-api-access-w2s8d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s2skg\" (UID: \"8531b165-d56f-4bdf-893a-b2395e9dd85e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.164866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.164797 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:35.537692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:35.537534 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg"] Nov 03 03:13:35.540376 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:13:35.540349 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8531b165_d56f_4bdf_893a_b2395e9dd85e.slice/crio-35f9ad48428c861744dcb00f592dc346cf8f0a38ce0748a448c47edf871ddd4b WatchSource:0}: Error finding container 35f9ad48428c861744dcb00f592dc346cf8f0a38ce0748a448c47edf871ddd4b: Status 404 returned error can't find the container with id 35f9ad48428c861744dcb00f592dc346cf8f0a38ce0748a448c47edf871ddd4b Nov 03 03:13:36.354150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:36.354104 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" event={"ID":"8531b165-d56f-4bdf-893a-b2395e9dd85e","Type":"ContainerStarted","Data":"74a327a03703bb88cd9a10fd506f3ba16f7530ac5c0dc352d3d06ef55cdb4ec4"} Nov 03 03:13:36.354537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:36.354159 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" event={"ID":"8531b165-d56f-4bdf-893a-b2395e9dd85e","Type":"ContainerStarted","Data":"35f9ad48428c861744dcb00f592dc346cf8f0a38ce0748a448c47edf871ddd4b"} Nov 03 03:13:37.165473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.165433 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:37.769901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.769834 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" podStartSLOduration=3.1878342 podStartE2EDuration="3.769817997s" totalImagesPullingTime="581.983797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:36.47572376 +0000 UTC m=+707.628412183" watchObservedRunningTime="2026-11-03 03:13:37.769817997 +0000 UTC m=+708.922506418" Nov 03 03:13:37.770944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.770926 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr"] Nov 03 03:13:37.774896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.774865 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.876812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.876780 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 03:13:37.878576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.878552 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr"] Nov 03 03:13:37.890697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.890677 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.890779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.890712 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.890779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.890732 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.891001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.890889 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.891090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.891010 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.891090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.891080 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99b8\" (UniqueName: \"kubernetes.io/projected/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-kube-api-access-k99b8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991552 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991609 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99b8\" (UniqueName: \"kubernetes.io/projected/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-kube-api-access-k99b8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991646 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991667 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991683 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.991810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.991703 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.992167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.992144 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.992227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.992149 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.992267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.992229 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.994210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.994189 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:37.994599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:37.994577 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:38.057268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:38.057229 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99b8\" (UniqueName: \"kubernetes.io/projected/a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e-kube-api-access-k99b8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr\" (UID: \"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:38.088054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:38.088015 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:38.467635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:38.467610 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr"] Nov 03 03:13:38.469539 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:13:38.469511 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cdd903_fe5b_4b17_b2b9_ee9e0535cd5e.slice/crio-7d16eb4897129a5f33f47d270cfcf51dfb6dc4a16f9c73141c349435a4d782bc WatchSource:0}: Error finding container 7d16eb4897129a5f33f47d270cfcf51dfb6dc4a16f9c73141c349435a4d782bc: Status 404 returned error can't find the container with id 7d16eb4897129a5f33f47d270cfcf51dfb6dc4a16f9c73141c349435a4d782bc Nov 03 03:13:39.374189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:39.374084 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" event={"ID":"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e","Type":"ContainerStarted","Data":"2c12c4e113999f00b2bf67cd318e518d556cbb8d8f18adfc7ac749da9a375561"} Nov 03 03:13:39.374189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:39.374151 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" event={"ID":"a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e","Type":"ContainerStarted","Data":"7d16eb4897129a5f33f47d270cfcf51dfb6dc4a16f9c73141c349435a4d782bc"} Nov 03 03:13:39.493860 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:39.493806 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" podStartSLOduration=1.9141948979999999 podStartE2EDuration="2.493786748s" totalImagesPullingTime="579.59185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:39.491314737 +0000 UTC m=+710.644003158" watchObservedRunningTime="2026-11-03 03:13:39.493786748 +0000 UTC m=+710.646475170" Nov 03 03:13:40.088146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:40.088104 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:45.165903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.165837 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:45.178883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.178840 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:45.420604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.420522 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s2skg" Nov 03 03:13:45.653813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.653783 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2"] Nov 03 03:13:45.657568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.657547 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.710568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.710490 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 03:13:45.721192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.721160 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2"] Nov 03 03:13:45.764328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764295 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.764540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764349 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.764596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764547 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.764629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764589 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6b7t\" (UniqueName: \"kubernetes.io/projected/ec639888-2e6f-40e9-899d-7409f7aef369-kube-api-access-t6b7t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.764678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.764712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.764690 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec639888-2e6f-40e9-899d-7409f7aef369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866103 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866139 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6b7t\" (UniqueName: \"kubernetes.io/projected/ec639888-2e6f-40e9-899d-7409f7aef369-kube-api-access-t6b7t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866184 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866209 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec639888-2e6f-40e9-899d-7409f7aef369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866274 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866321 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866597 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866694 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.866909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.866838 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.868807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.868777 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec639888-2e6f-40e9-899d-7409f7aef369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.869196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.869178 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec639888-2e6f-40e9-899d-7409f7aef369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.928789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.928750 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6b7t\" (UniqueName: \"kubernetes.io/projected/ec639888-2e6f-40e9-899d-7409f7aef369-kube-api-access-t6b7t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2\" (UID: \"ec639888-2e6f-40e9-899d-7409f7aef369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:45.971515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:45.971425 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:46.405508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:46.404998 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2"] Nov 03 03:13:46.407536 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:13:46.407501 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec639888_2e6f_40e9_899d_7409f7aef369.slice/crio-6580c765974a33ec765acc73ad20b52f1b6fc5293801c5ec50edf54438223b6e WatchSource:0}: Error finding container 6580c765974a33ec765acc73ad20b52f1b6fc5293801c5ec50edf54438223b6e: Status 404 returned error can't find the container with id 6580c765974a33ec765acc73ad20b52f1b6fc5293801c5ec50edf54438223b6e Nov 03 03:13:46.416250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:46.416213 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" event={"ID":"ec639888-2e6f-40e9-899d-7409f7aef369","Type":"ContainerStarted","Data":"6580c765974a33ec765acc73ad20b52f1b6fc5293801c5ec50edf54438223b6e"} Nov 03 03:13:47.424293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:47.424196 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" event={"ID":"ec639888-2e6f-40e9-899d-7409f7aef369","Type":"ContainerStarted","Data":"b0eb4088bde4f485af6132932a6d16799172312cc71ef1709953f5afaa84ac6f"} Nov 03 03:13:47.544489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:47.544428 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" podStartSLOduration=1.9175378859999999 podStartE2EDuration="2.544411524s" totalImagesPullingTime="626.873638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:47.543416011 +0000 UTC m=+718.696104436" watchObservedRunningTime="2026-11-03 03:13:47.544411524 +0000 UTC m=+718.697099944" Nov 03 03:13:47.972297 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:47.972253 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:48.088340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:48.088302 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:48.100959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:48.100928 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:48.440311 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:48.440280 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr" Nov 03 03:13:55.972421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:55.972319 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:55.991975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:55.991938 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:13:56.486643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:13:56.486613 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2" Nov 03 03:14:02.329022 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:14:02.328983 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:14:19.688747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.688702 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c66d77cd-926q9"] Nov 03 03:14:19.692973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.692949 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.748619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.748590 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c66d77cd-926q9"] Nov 03 03:14:19.785866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.785801 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178bdf09-2334-43af-93f3-175fc348c04a-tls-cert\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.786360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.786337 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ck5r\" (UniqueName: \"kubernetes.io/projected/178bdf09-2334-43af-93f3-175fc348c04a-kube-api-access-7ck5r\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.887389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.887352 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178bdf09-2334-43af-93f3-175fc348c04a-tls-cert\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.887570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.887452 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ck5r\" (UniqueName: \"kubernetes.io/projected/178bdf09-2334-43af-93f3-175fc348c04a-kube-api-access-7ck5r\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.889954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.889928 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/178bdf09-2334-43af-93f3-175fc348c04a-tls-cert\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:19.948498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:19.948432 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ck5r\" (UniqueName: \"kubernetes.io/projected/178bdf09-2334-43af-93f3-175fc348c04a-kube-api-access-7ck5r\") pod \"authorino-56c66d77cd-926q9\" (UID: \"178bdf09-2334-43af-93f3-175fc348c04a\") " pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:20.007633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:20.007593 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c66d77cd-926q9" Nov 03 03:14:20.377566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:20.377515 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c66d77cd-926q9"] Nov 03 03:14:20.379999 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:14:20.379970 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod178bdf09_2334_43af_93f3_175fc348c04a.slice/crio-fdcb835fce03e43fccaa7b31f5a78aaee36abddc26988589eb7a7111d93c1322 WatchSource:0}: Error finding container fdcb835fce03e43fccaa7b31f5a78aaee36abddc26988589eb7a7111d93c1322: Status 404 returned error can't find the container with id fdcb835fce03e43fccaa7b31f5a78aaee36abddc26988589eb7a7111d93c1322 Nov 03 03:14:20.600664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:20.600622 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c66d77cd-926q9" event={"ID":"178bdf09-2334-43af-93f3-175fc348c04a","Type":"ContainerStarted","Data":"fdcb835fce03e43fccaa7b31f5a78aaee36abddc26988589eb7a7111d93c1322"} Nov 03 03:14:22.612813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:22.612772 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c66d77cd-926q9" event={"ID":"178bdf09-2334-43af-93f3-175fc348c04a","Type":"ContainerStarted","Data":"6292eab11ce7b85f9dd6fc1254595803c947c7ab4dd93c824bbd03786c413900"} Nov 03 03:14:22.727889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:22.727813 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c66d77cd-926q9" podStartSLOduration=2.214395755 podStartE2EDuration="3.727788496s" totalImagesPullingTime="1.513392741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:22.727777621 +0000 UTC m=+753.880466046" watchObservedRunningTime="2026-11-03 03:14:22.727788496 +0000 UTC m=+753.880476917" Nov 03 03:14:22.759613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:22.759577 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:14:22.759822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:22.759788 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-64797fcd5-mmjpq" podUID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" containerName="authorino" containerID="cri-o://3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b" gracePeriod=30 Nov 03 03:14:23.024226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.024189 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:14:23.117279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.117244 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\" (UniqueName: \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\") pod \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " Nov 03 03:14:23.117478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.117356 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\" (UniqueName: \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\") pod \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\" (UID: \"da7e79c5-6cbc-49d7-b4c4-b531c930f98d\") " Nov 03 03:14:23.119632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.119599 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99" pod "da7e79c5-6cbc-49d7-b4c4-b531c930f98d" (UID: "da7e79c5-6cbc-49d7-b4c4-b531c930f98d"). InnerVolumeSpecName "kube-api-access-7mq99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:23.127948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.127916 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert" pod "da7e79c5-6cbc-49d7-b4c4-b531c930f98d" (UID: "da7e79c5-6cbc-49d7-b4c4-b531c930f98d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:23.218906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.218867 5275 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-tls-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:14:23.218906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.218899 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mq99\" (UniqueName: \"kubernetes.io/projected/da7e79c5-6cbc-49d7-b4c4-b531c930f98d-kube-api-access-7mq99\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:14:23.618997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.618958 5275 generic.go:356] "Generic (PLEG): container finished" podID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" containerID="3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b" exitCode=0 Nov 03 03:14:23.619488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.619007 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64797fcd5-mmjpq" Nov 03 03:14:23.619488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.619006 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64797fcd5-mmjpq" event={"ID":"da7e79c5-6cbc-49d7-b4c4-b531c930f98d","Type":"ContainerDied","Data":"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b"} Nov 03 03:14:23.619488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.619060 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64797fcd5-mmjpq" event={"ID":"da7e79c5-6cbc-49d7-b4c4-b531c930f98d","Type":"ContainerDied","Data":"efcbaf4b6eafe5bd5a31679ebe962fa396ce7d222c57f70ef871679ce173b79c"} Nov 03 03:14:23.619488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.619085 5275 scope.go:122] "RemoveContainer" containerID="3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b" Nov 03 03:14:23.630436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.630414 5275 scope.go:122] "RemoveContainer" containerID="3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b" Nov 03 03:14:23.630775 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:14:23.630756 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b\": container with ID starting with 3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b not found: ID does not exist" containerID="3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b" Nov 03 03:14:23.630835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.630788 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b"} err="failed to get container status \"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b\": rpc error: code = NotFound desc = could not find container \"3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b\": container with ID starting with 3200b56a3ddd1c50cdfcd301e59dbdd20af7cc4b7f6fd429c3677c52a40cde1b not found: ID does not exist" Nov 03 03:14:23.782691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.782661 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:14:23.831628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:23.831580 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64797fcd5-mmjpq"] Nov 03 03:14:24.983895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:24.983841 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" path="/var/lib/kubelet/pods/da7e79c5-6cbc-49d7-b4c4-b531c930f98d/volumes" Nov 03 03:14:31.419674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.419640 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:14:31.420508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.420074 5275 cpu_manager.go:414] "containerMap: removing container" podUID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" containerName="authorino" Nov 03 03:14:31.420508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.420088 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" containerName="authorino" Nov 03 03:14:31.420508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.420155 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7e79c5-6cbc-49d7-b4c4-b531c930f98d" containerName="authorino" Nov 03 03:14:31.423358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.423338 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.518818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.518785 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.518832 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.518872 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519020 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519051 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5tk\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519075 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519091 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519199 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.519319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.519288 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.527934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.527905 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:14:31.619928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.619893 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.619928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.619928 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.619951 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.619994 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620145 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620201 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620238 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620344 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620376 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5tk\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620394 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620663 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.620973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.620945 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.621125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.621061 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.622942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.622917 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.623749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.623728 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.682454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.682381 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.682454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.682439 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5tk\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\") pod \"maas-default-gateway-openshift-default-76f4876667-j4jzb\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:31.736981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:31.736946 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:32.175317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.175264 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:14:32.177637 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:14:32.177606 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1833e32_0f17_4659_a75b_4c490cec5c7f.slice/crio-febe7aae4051bb6d0f6910da6fbb4f26a224fc6c20d4736253402087d481dce9 WatchSource:0}: Error finding container febe7aae4051bb6d0f6910da6fbb4f26a224fc6c20d4736253402087d481dce9: Status 404 returned error can't find the container with id febe7aae4051bb6d0f6910da6fbb4f26a224fc6c20d4736253402087d481dce9 Nov 03 03:14:32.179572 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.179538 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:32.179656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.179618 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:32.179656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.179651 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:32.334685 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:14:32.334623 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:14:32.674309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.674270 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" event={"ID":"c1833e32-0f17-4659-a75b-4c490cec5c7f","Type":"ContainerStarted","Data":"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11"} Nov 03 03:14:32.674911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.674884 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" event={"ID":"c1833e32-0f17-4659-a75b-4c490cec5c7f","Type":"ContainerStarted","Data":"febe7aae4051bb6d0f6910da6fbb4f26a224fc6c20d4736253402087d481dce9"} Nov 03 03:14:32.737321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.737295 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:32.798380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:32.798326 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podStartSLOduration=1.7983120750000001 podStartE2EDuration="1.798312075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:32.795678363 +0000 UTC m=+763.948366784" watchObservedRunningTime="2026-11-03 03:14:32.798312075 +0000 UTC m=+763.951000496" Nov 03 03:14:33.738695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:33.738655 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:33.739192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:33.738717 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:34.737611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:34.737565 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:34.737816 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:34.737641 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:35.737919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:35.737881 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:35.738398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:35.737954 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:36.737318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:36.737282 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:36.737499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:36.737343 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:37.738305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:37.738266 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:37.738704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:37.738319 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:38.738104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:38.738057 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:38.738324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:38.738118 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:39.737708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:39.737668 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 03:14:39.737945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:39.737731 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 03:14:40.742095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:40.742067 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:40.742501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:40.742399 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:40.742958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:40.742938 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:14:59.114369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.114338 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:14:59.117912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.117891 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.176822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.176792 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e21b4958-openshift-default-dockercfg-r8fzr\"" Nov 03 03:14:59.208252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208218 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208279 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208340 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208397 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208462 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208490 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208512 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208530 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrns\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.208636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.208572 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.225319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.225283 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:14:59.309178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309146 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309186 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309223 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309251 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrns\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309278 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309358 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309414 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309445 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309521 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309565 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309647 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.309994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.309968 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.310060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.310034 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.310595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.310569 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.312110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.312087 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.312482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.312463 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.370977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.370908 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.371099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.371038 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrns\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\") pod \"e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.429941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.429913 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:14:59.745005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.744926 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:14:59.749730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.749707 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.802898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.802227 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4fb905b5-gw-openshift-default-dockercfg-4hxsc\"" Nov 03 03:14:59.815412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.815225 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:14:59.815775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.815751 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.815927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.815809 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5gc\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.815927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.815889 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.815954 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.816002 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.816053 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.816148 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.816212 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.816317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.816254 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.904172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.904146 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:14:59.905620 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:14:59.905591 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84db5af4_e74c_4507_aff5_675fdc33639d.slice/crio-287af8def80fbb92d2babcd1e2f061ddfcf369dd41a1d996bd8e6a74fe99188d WatchSource:0}: Error finding container 287af8def80fbb92d2babcd1e2f061ddfcf369dd41a1d996bd8e6a74fe99188d: Status 404 returned error can't find the container with id 287af8def80fbb92d2babcd1e2f061ddfcf369dd41a1d996bd8e6a74fe99188d Nov 03 03:14:59.907896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.907859 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:59.907989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.907928 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:59.907989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.907960 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:14:59.917135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917113 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917152 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917280 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917328 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5gc\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917396 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917421 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917466 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917515 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.917599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917557 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.918112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917770 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.918112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917788 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.918112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.917892 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.918112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.918016 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.918112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.918045 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.919954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.919939 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.920085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.920069 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.978426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.978400 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5gc\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:14:59.978555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:14:59.978531 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\") pod \"e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:00.062642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.062611 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:00.173326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.173289 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:00.178597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.178574 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.221520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.221483 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g567v\" (UniqueName: \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.221687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.221553 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.231651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.231614 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:00.231798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.231669 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:00.232449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.232428 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:15:00.234736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.234714 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:00.322493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.322404 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.322639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.322517 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g567v\" (UniqueName: \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.323200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.323175 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.392905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.392866 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g567v\" (UniqueName: \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\") pod \"maas-api-key-cleanup-29894595-jdfm7\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.492065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.492038 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:00.499013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.498988 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:15:00.502317 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:00.502287 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf5ad3d_e8d1_46bb_9f65_769b87057ccf.slice/crio-d6602c482075ad0ff9a4033689044ffdbb9c19d06a7579f45f42bf65bdc19819 WatchSource:0}: Error finding container d6602c482075ad0ff9a4033689044ffdbb9c19d06a7579f45f42bf65bdc19819: Status 404 returned error can't find the container with id d6602c482075ad0ff9a4033689044ffdbb9c19d06a7579f45f42bf65bdc19819 Nov 03 03:15:00.505327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.505289 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:00.505423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.505402 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:00.505472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.505453 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:00.843659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.843553 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerStarted","Data":"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4"} Nov 03 03:15:00.843659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.843626 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerStarted","Data":"d6602c482075ad0ff9a4033689044ffdbb9c19d06a7579f45f42bf65bdc19819"} Nov 03 03:15:00.845327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.845300 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerStarted","Data":"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed"} Nov 03 03:15:00.845441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.845354 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerStarted","Data":"287af8def80fbb92d2babcd1e2f061ddfcf369dd41a1d996bd8e6a74fe99188d"} Nov 03 03:15:00.877252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.877225 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:00.878971 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:00.878948 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edb3b6a_ff74_477e_87c1_4b5fb94b41c8.slice/crio-855b628a1962b1713ef154e67f8e4271df147e58b7e28763b23502e0b66a0e70 WatchSource:0}: Error finding container 855b628a1962b1713ef154e67f8e4271df147e58b7e28763b23502e0b66a0e70: Status 404 returned error can't find the container with id 855b628a1962b1713ef154e67f8e4271df147e58b7e28763b23502e0b66a0e70 Nov 03 03:15:00.966594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:00.966526 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podStartSLOduration=1.9665114209999999 podStartE2EDuration="1.966511421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:00.965068763 +0000 UTC m=+792.117757185" watchObservedRunningTime="2026-11-03 03:15:00.966511421 +0000 UTC m=+792.119200170" Nov 03 03:15:01.063266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.063234 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:01.068445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.068418 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:01.090037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.089970 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podStartSLOduration=2.08995383 podStartE2EDuration="2.08995383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:01.087721894 +0000 UTC m=+792.240410390" watchObservedRunningTime="2026-11-03 03:15:01.08995383 +0000 UTC m=+792.242642251" Nov 03 03:15:01.430171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.430125 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:01.435741 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.435704 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:01.852701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.852663 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerStarted","Data":"855b628a1962b1713ef154e67f8e4271df147e58b7e28763b23502e0b66a0e70"} Nov 03 03:15:01.853145 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.853114 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:01.853302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.853164 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:01.853819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.853797 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:01.854024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:01.854009 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:02.329324 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:15:02.329289 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:15:05.880752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:05.880713 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerStarted","Data":"4824800298fa2dc75a4420d56f7dae66961f0decf38bdd668b51b7c98eb85081"} Nov 03 03:15:05.996985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:05.996908 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" podStartSLOduration=1.859043402 podStartE2EDuration="5.99688867s" totalImagesPullingTime="4.137845268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:05.995966074 +0000 UTC m=+797.148654496" watchObservedRunningTime="2026-11-03 03:15:05.99688867 +0000 UTC m=+797.149577093" Nov 03 03:15:06.887294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:06.887260 5275 generic.go:356] "Generic (PLEG): container finished" podID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerID="4824800298fa2dc75a4420d56f7dae66961f0decf38bdd668b51b7c98eb85081" exitCode=60 Nov 03 03:15:06.887716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:06.887343 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerDied","Data":"4824800298fa2dc75a4420d56f7dae66961f0decf38bdd668b51b7c98eb85081"} Nov 03 03:15:06.887716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:06.887554 5275 scope.go:122] "RemoveContainer" containerID="4824800298fa2dc75a4420d56f7dae66961f0decf38bdd668b51b7c98eb85081" Nov 03 03:15:07.238459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.238427 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:15:07.242063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.242041 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294595 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294627 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294652 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294797 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294881 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wdb\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.294979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.294952 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.295052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.295029 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.295087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.295050 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.295137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.295088 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.299224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.299193 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-69e611c0-openshift-default-dockercfg-dkr7b\"" Nov 03 03:15:07.310978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.310948 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:15:07.396450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396415 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396455 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396578 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396654 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396682 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396720 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396761 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396830 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wdb\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.396915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.396887 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.397363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.397077 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.397363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.397243 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.397363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.397272 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.397363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.397315 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.397612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.397508 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.399105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.399082 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.399212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.399194 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.461461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.461386 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wdb\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.461461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.461433 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\") pod \"e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.556904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.556257 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:07.896869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.896808 5275 generic.go:356] "Generic (PLEG): container finished" podID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerID="66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75" exitCode=60 Nov 03 03:15:07.897320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.896897 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerDied","Data":"66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75"} Nov 03 03:15:07.897320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.896954 5275 scope.go:122] "RemoveContainer" containerID="4824800298fa2dc75a4420d56f7dae66961f0decf38bdd668b51b7c98eb85081" Nov 03 03:15:07.897320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.897204 5275 scope.go:122] "RemoveContainer" containerID="66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75" Nov 03 03:15:07.897492 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:15:07.897473 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-jdfm7_odh-ai-gateway-infra(2edb3b6a-ff74-477e-87c1-4b5fb94b41c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" Nov 03 03:15:07.989903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.989835 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:15:07.991221 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:07.991198 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13791e77_2f7c_432a_8845_155113624cc3.slice/crio-606f0d550128f3265216fdb013a2d7e49656d443e60fee423f309f52b56aad0e WatchSource:0}: Error finding container 606f0d550128f3265216fdb013a2d7e49656d443e60fee423f309f52b56aad0e: Status 404 returned error can't find the container with id 606f0d550128f3265216fdb013a2d7e49656d443e60fee423f309f52b56aad0e Nov 03 03:15:07.993599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.993566 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:07.993671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.993634 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:07.993671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:07.993661 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:08.904160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:08.904130 5275 scope.go:122] "RemoveContainer" containerID="66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75" Nov 03 03:15:08.904622 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:15:08.904405 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-jdfm7_odh-ai-gateway-infra(2edb3b6a-ff74-477e-87c1-4b5fb94b41c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" Nov 03 03:15:08.905514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:08.905487 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerStarted","Data":"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c"} Nov 03 03:15:08.905653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:08.905532 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerStarted","Data":"606f0d550128f3265216fdb013a2d7e49656d443e60fee423f309f52b56aad0e"} Nov 03 03:15:09.150061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:09.150006 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podStartSLOduration=2.149989766 podStartE2EDuration="2.149989766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:09.147826104 +0000 UTC m=+800.300514541" watchObservedRunningTime="2026-11-03 03:15:09.149989766 +0000 UTC m=+800.302678187" Nov 03 03:15:09.557229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:09.557185 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:09.562194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:09.562150 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:09.910906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:09.910819 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:09.911784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:09.911767 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:12.707087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.707052 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:15:12.710641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.710624 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.756208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.756167 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkss\" (UniqueName: \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.756365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.756327 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.820874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.820819 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e21b4958-dockercfg-vnrcq\"" Nov 03 03:15:12.821076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.820991 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e21b4958\"" Nov 03 03:15:12.823324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.823295 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:15:12.851784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.851745 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:15:12.855990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.855970 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:12.857452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.857421 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.857593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.857557 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkss\" (UniqueName: \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.858218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.858194 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.914961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.914926 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:15:12.919610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.919574 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkss\" (UniqueName: \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\") pod \"payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:12.958647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.958549 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlc7\" (UniqueName: \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:12.958647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:12.958607 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.026158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.026127 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:13.059313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.059277 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.059887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.059544 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlc7\" (UniqueName: \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.060047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.060027 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.122119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.122087 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlc7\" (UniqueName: \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\") pod \"payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.171043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.171008 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:13.404825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.404796 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:15:13.405932 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:13.405908 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f235e68_b3f9_4603_be50_fef9e0ab6366.slice/crio-2646c3eba11d9fb8fc80adef9aeb7cd4647e820b0a05d8615cd3d62e149361fd WatchSource:0}: Error finding container 2646c3eba11d9fb8fc80adef9aeb7cd4647e820b0a05d8615cd3d62e149361fd: Status 404 returned error can't find the container with id 2646c3eba11d9fb8fc80adef9aeb7cd4647e820b0a05d8615cd3d62e149361fd Nov 03 03:15:13.545686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.545635 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:15:13.547877 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:13.547840 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6653ef49_967b_41e6_80b1_0f60385c74df.slice/crio-edea2372867e0262c057b43c9f121af79ce9e2b8378632348bce969ed8cdcb19 WatchSource:0}: Error finding container edea2372867e0262c057b43c9f121af79ce9e2b8378632348bce969ed8cdcb19: Status 404 returned error can't find the container with id edea2372867e0262c057b43c9f121af79ce9e2b8378632348bce969ed8cdcb19 Nov 03 03:15:13.934080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.933993 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" event={"ID":"9f235e68-b3f9-4603-be50-fef9e0ab6366","Type":"ContainerStarted","Data":"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315"} Nov 03 03:15:13.934080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.934048 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" event={"ID":"9f235e68-b3f9-4603-be50-fef9e0ab6366","Type":"ContainerStarted","Data":"2646c3eba11d9fb8fc80adef9aeb7cd4647e820b0a05d8615cd3d62e149361fd"} Nov 03 03:15:13.934564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.934150 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:13.935647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.935623 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" event={"ID":"6653ef49-967b-41e6-80b1-0f60385c74df","Type":"ContainerStarted","Data":"87a77a4e7e889ae122eda676335f6b808022975679fe531b6a70ba6a71685472"} Nov 03 03:15:13.935762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.935663 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" event={"ID":"6653ef49-967b-41e6-80b1-0f60385c74df","Type":"ContainerStarted","Data":"edea2372867e0262c057b43c9f121af79ce9e2b8378632348bce969ed8cdcb19"} Nov 03 03:15:13.935762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:13.935752 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:14.051144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:14.051094 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" podStartSLOduration=2.051078389 podStartE2EDuration="2.051078389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:14.049440434 +0000 UTC m=+805.202128875" watchObservedRunningTime="2026-11-03 03:15:14.051078389 +0000 UTC m=+805.203766809" Nov 03 03:15:14.167680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:14.167615 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" podStartSLOduration=2.167593916 podStartE2EDuration="2.167593916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:14.16446166 +0000 UTC m=+805.317150080" watchObservedRunningTime="2026-11-03 03:15:14.167593916 +0000 UTC m=+805.320282339" Nov 03 03:15:15.332574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.332543 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:15:15.336443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.336425 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.394336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.394308 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:15:15.440482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.440452 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b4ff4779-dockercfg-n5gjj\"" Nov 03 03:15:15.441415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.441387 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b4ff4779\"" Nov 03 03:15:15.484325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.484291 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgks2\" (UniqueName: \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.484527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.484433 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.586093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.586010 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.586240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.586112 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgks2\" (UniqueName: \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.586782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.586759 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.647263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.647227 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgks2\" (UniqueName: \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\") pod \"payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:15.674176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:15.674141 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:16.054048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:16.054025 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:15:16.056158 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:16.056130 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd386991_cb34_4776_8b3b_ea2e7dd63b5e.slice/crio-f1073ff11b018fd684f67d401744f9c327c7438a5afd3769c740ac0124cb6e6c WatchSource:0}: Error finding container f1073ff11b018fd684f67d401744f9c327c7438a5afd3769c740ac0124cb6e6c: Status 404 returned error can't find the container with id f1073ff11b018fd684f67d401744f9c327c7438a5afd3769c740ac0124cb6e6c Nov 03 03:15:16.954424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:16.954394 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" event={"ID":"bd386991-cb34-4776-8b3b-ea2e7dd63b5e","Type":"ContainerStarted","Data":"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc"} Nov 03 03:15:16.954914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:16.954466 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" event={"ID":"bd386991-cb34-4776-8b3b-ea2e7dd63b5e","Type":"ContainerStarted","Data":"f1073ff11b018fd684f67d401744f9c327c7438a5afd3769c740ac0124cb6e6c"} Nov 03 03:15:16.954914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:16.954501 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:17.075502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.075446 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" podStartSLOduration=2.075430512 podStartE2EDuration="2.075430512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:17.07302035 +0000 UTC m=+808.225708796" watchObservedRunningTime="2026-11-03 03:15:17.075430512 +0000 UTC m=+808.228118990" Nov 03 03:15:17.868821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.868786 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:15:17.873104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.873077 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:17.927555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.927520 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-56094c67\"" Nov 03 03:15:17.930625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.930587 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:15:17.982180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:17.982154 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-56094c67-dockercfg-mhz42\"" Nov 03 03:15:18.010583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.010551 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk67h\" (UniqueName: \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.010782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.010620 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.111927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.111883 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk67h\" (UniqueName: \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.112128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.112011 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.112842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.112819 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.173671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.173587 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk67h\" (UniqueName: \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\") pod \"payload-processing-e2e-ait-56094c67-67598699f9-v84ng\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.187423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.187393 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:18.572057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.572024 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:15:18.573048 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:18.573023 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49341fdc_5e87_4947_9847_0f86708eb515.slice/crio-179b6dc45d0a094a5ee57b1e164691a01da145b19b73937819891fab02f1958d WatchSource:0}: Error finding container 179b6dc45d0a094a5ee57b1e164691a01da145b19b73937819891fab02f1958d: Status 404 returned error can't find the container with id 179b6dc45d0a094a5ee57b1e164691a01da145b19b73937819891fab02f1958d Nov 03 03:15:18.968422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.968318 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" event={"ID":"49341fdc-5e87-4947-9847-0f86708eb515","Type":"ContainerStarted","Data":"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61"} Nov 03 03:15:18.968422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.968374 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" event={"ID":"49341fdc-5e87-4947-9847-0f86708eb515","Type":"ContainerStarted","Data":"179b6dc45d0a094a5ee57b1e164691a01da145b19b73937819891fab02f1958d"} Nov 03 03:15:18.968422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:18.968406 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:19.084242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:19.084173 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" podStartSLOduration=2.0841561730000002 podStartE2EDuration="2.084156173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:19.08375861 +0000 UTC m=+810.236447032" watchObservedRunningTime="2026-11-03 03:15:19.084156173 +0000 UTC m=+810.236844596" Nov 03 03:15:19.977973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:19.977943 5275 scope.go:122] "RemoveContainer" containerID="66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75" Nov 03 03:15:20.981063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:20.981014 5275 generic.go:356] "Generic (PLEG): container finished" podID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerID="ebecaefdd28758347a00c4017539b7ec226510809b5531fcdabc5fdfe836888b" exitCode=60 Nov 03 03:15:20.984748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:20.984713 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerDied","Data":"ebecaefdd28758347a00c4017539b7ec226510809b5531fcdabc5fdfe836888b"} Nov 03 03:15:20.984941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:20.984787 5275 scope.go:122] "RemoveContainer" containerID="66d12b95e9cffe9f9f3c22af3c0338a0bdce02c3fac9b474f395f7e80aae0a75" Nov 03 03:15:20.985228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:20.985205 5275 scope.go:122] "RemoveContainer" containerID="ebecaefdd28758347a00c4017539b7ec226510809b5531fcdabc5fdfe836888b" Nov 03 03:15:20.985540 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:15:20.985516 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-jdfm7_odh-ai-gateway-infra(2edb3b6a-ff74-477e-87c1-4b5fb94b41c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" Nov 03 03:15:22.114536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.114491 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:22.253564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.253538 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:22.355172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.355135 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\") pod \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " Nov 03 03:15:22.355345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.355278 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\" (UniqueName: \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\") pod \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\" (UID: \"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8\") " Nov 03 03:15:22.355530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.355505 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca" pod "2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" (UID: "2edb3b6a-ff74-477e-87c1-4b5fb94b41c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:22.357763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.357730 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v" pod "2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" (UID: "2edb3b6a-ff74-477e-87c1-4b5fb94b41c8"). InnerVolumeSpecName "kube-api-access-g567v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:22.456745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.456642 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g567v\" (UniqueName: \"kubernetes.io/projected/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-kube-api-access-g567v\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:22.456745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.456667 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:22.995168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.995134 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" Nov 03 03:15:22.995370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.995181 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7" event={"ID":"2edb3b6a-ff74-477e-87c1-4b5fb94b41c8","Type":"ContainerDied","Data":"855b628a1962b1713ef154e67f8e4271df147e58b7e28763b23502e0b66a0e70"} Nov 03 03:15:22.995370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:22.995255 5275 scope.go:122] "RemoveContainer" containerID="ebecaefdd28758347a00c4017539b7ec226510809b5531fcdabc5fdfe836888b" Nov 03 03:15:23.166873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:23.163489 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:23.213466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:23.213423 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-jdfm7"] Nov 03 03:15:24.944705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:24.944676 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:15:24.945137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:24.944739 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:15:24.984426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:24.984391 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" path="/var/lib/kubelet/pods/2edb3b6a-ff74-477e-87c1-4b5fb94b41c8/volumes" Nov 03 03:15:27.200737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.200702 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:15:27.201183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201133 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.201183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201146 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.201183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201173 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.201183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201178 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.201315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201246 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.201315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.201254 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:27.204387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.204368 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.310693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.310656 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-69e611c0\"" Nov 03 03:15:27.310693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.310656 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-69e611c0-dockercfg-g974c\"" Nov 03 03:15:27.313035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.313014 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:15:27.328193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.328156 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.328382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.328328 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zx2\" (UniqueName: \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.429377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.429336 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zx2\" (UniqueName: \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.429549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.429457 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.430505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.430479 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.490959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.490876 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zx2\" (UniqueName: \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\") pod \"payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.517361 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.517328 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:27.916508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.916484 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:15:27.918272 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:27.918248 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf42cf6_7d10_493c_91c6_3c2c1960c805.slice/crio-6ed3c6881d6448b0205f780d25cae6f0b0cd88d0ce00e37330ddedbe7d0bb0f6 WatchSource:0}: Error finding container 6ed3c6881d6448b0205f780d25cae6f0b0cd88d0ce00e37330ddedbe7d0bb0f6: Status 404 returned error can't find the container with id 6ed3c6881d6448b0205f780d25cae6f0b0cd88d0ce00e37330ddedbe7d0bb0f6 Nov 03 03:15:27.966392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:27.966370 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:15:28.026718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:28.026686 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" event={"ID":"caf42cf6-7d10-493c-91c6-3c2c1960c805","Type":"ContainerStarted","Data":"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff"} Nov 03 03:15:28.026898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:28.026741 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:28.026898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:28.026756 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" event={"ID":"caf42cf6-7d10-493c-91c6-3c2c1960c805","Type":"ContainerStarted","Data":"6ed3c6881d6448b0205f780d25cae6f0b0cd88d0ce00e37330ddedbe7d0bb0f6"} Nov 03 03:15:28.255442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:28.255329 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" podStartSLOduration=1.255314242 podStartE2EDuration="1.255314242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:28.253000265 +0000 UTC m=+819.405688686" watchObservedRunningTime="2026-11-03 03:15:28.255314242 +0000 UTC m=+819.408002675" Nov 03 03:15:29.976569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:29.976540 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:15:31.854159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:31.854120 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:31.854686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:31.854177 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:32.050641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.050609 5275 generic.go:356] "Generic (PLEG): container finished" podID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerID="298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4" exitCode=0 Nov 03 03:15:32.050836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.050690 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerDied","Data":"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4"} Nov 03 03:15:32.051171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.051154 5275 scope.go:122] "RemoveContainer" containerID="298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4" Nov 03 03:15:32.052485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.052457 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:32.052545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.052526 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:32.052581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:32.052558 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:32.332410 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:15:32.332369 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:15:33.081469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:33.081434 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerStarted","Data":"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995"} Nov 03 03:15:34.069203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:34.069168 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:34.069561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:34.069480 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:34.069734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:34.069709 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:35.072008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:35.071974 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:35.072633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:35.072591 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:36.069743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:36.069292 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:36.069743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:36.069356 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:37.067299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:37.067262 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:37.067941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:37.067906 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:38.063587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:38.063545 5275 patch_prober.go:29] interesting pod/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 03:15:38.063770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:38.063616 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 03:15:39.035804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:39.035759 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:15:39.068070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:39.068031 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:39.068351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:39.068327 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:39.069233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:39.069210 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:15:47.895045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:47.894997 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:15:47.895543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:47.895079 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:15:49.185877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.185763 5275 generic.go:356] "Generic (PLEG): container finished" podID="84db5af4-e74c-4507-aff5-675fdc33639d" containerID="e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed" exitCode=0 Nov 03 03:15:49.186515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.185925 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerDied","Data":"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed"} Nov 03 03:15:49.187021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.186931 5275 scope.go:122] "RemoveContainer" containerID="e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed" Nov 03 03:15:49.189927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.189214 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:49.189927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.189316 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:49.189927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:49.189360 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:50.201209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.201133 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerStarted","Data":"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094"} Nov 03 03:15:50.431375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.431335 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:50.431637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.431613 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 03:15:50.431812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.431763 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 03:15:50.775877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.773705 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7"] Nov 03 03:15:50.783871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.782494 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:50.783871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.782525 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:50.783871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.782665 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2edb3b6a-ff74-477e-87c1-4b5fb94b41c8" containerName="cleanup" Nov 03 03:15:50.797874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.796263 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.816798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.816765 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.816972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.816948 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dkr\" (UniqueName: \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.817126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.816975 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.839749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.839720 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7"] Nov 03 03:15:50.869887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.849037 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:15:50.869887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.853090 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:50.869887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.853157 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:15:50.869887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.853241 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:50.918474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.918437 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.918683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.918486 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.918683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.918585 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dkr\" (UniqueName: \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.919501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.919473 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:50.921017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:50.920997 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:51.002949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:51.002912 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dkr\" (UniqueName: \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\") pod \"maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:51.114746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:51.114663 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:51.430427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:51.430345 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 03:15:51.430427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:51.430403 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 03:15:51.539642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:51.539613 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7"] Nov 03 03:15:51.542834 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:15:51.542044 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a17ed26_e15b_483a_ac57_154345b63758.slice/crio-7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5 WatchSource:0}: Error finding container 7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5: Status 404 returned error can't find the container with id 7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5 Nov 03 03:15:52.215998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:52.215911 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" event={"ID":"1a17ed26-e15b-483a-ac57-154345b63758","Type":"ContainerStarted","Data":"1bd66bd2284bedb6e4b6bfb9728649ccd10ec89b203685b8a0cba7ece05112c0"} Nov 03 03:15:52.216317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:52.216271 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" event={"ID":"1a17ed26-e15b-483a-ac57-154345b63758","Type":"ContainerStarted","Data":"7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5"} Nov 03 03:15:52.336124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:52.336058 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" podStartSLOduration=2.33603658 podStartE2EDuration="2.33603658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:52.333796227 +0000 UTC m=+843.486484662" watchObservedRunningTime="2026-11-03 03:15:52.33603658 +0000 UTC m=+843.488725002" Nov 03 03:15:52.430335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:52.430299 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 03:15:52.430575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:52.430554 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 03:15:53.222512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:53.222423 5275 generic.go:356] "Generic (PLEG): container finished" podID="1a17ed26-e15b-483a-ac57-154345b63758" containerID="1bd66bd2284bedb6e4b6bfb9728649ccd10ec89b203685b8a0cba7ece05112c0" exitCode=0 Nov 03 03:15:53.222655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:53.222524 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" event={"ID":"1a17ed26-e15b-483a-ac57-154345b63758","Type":"ContainerDied","Data":"1bd66bd2284bedb6e4b6bfb9728649ccd10ec89b203685b8a0cba7ece05112c0"} Nov 03 03:15:53.430368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:53.430335 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 03:15:53.430535 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:53.430389 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 03:15:54.318667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.318633 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk"] Nov 03 03:15:54.323197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.323163 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.433372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.429292 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.433372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.429348 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.433372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.429459 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8nj\" (UniqueName: \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.433372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.430738 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 03:15:54.433372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.430785 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 03:15:54.435932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.435884 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk"] Nov 03 03:15:54.535455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.534616 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.535455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.534655 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.535455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.534754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8nj\" (UniqueName: \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.535455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.535379 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.542231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.542177 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.636385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.636319 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8nj\" (UniqueName: \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\") pod \"maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.639997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.639007 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:54.704687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.704439 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:54.841687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.841654 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\") pod \"1a17ed26-e15b-483a-ac57-154345b63758\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " Nov 03 03:15:54.841865 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.841720 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\" (UniqueName: \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\") pod \"1a17ed26-e15b-483a-ac57-154345b63758\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " Nov 03 03:15:54.841865 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.841762 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\") pod \"1a17ed26-e15b-483a-ac57-154345b63758\" (UID: \"1a17ed26-e15b-483a-ac57-154345b63758\") " Nov 03 03:15:54.842675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.842652 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca" pod "1a17ed26-e15b-483a-ac57-154345b63758" (UID: "1a17ed26-e15b-483a-ac57-154345b63758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:54.845514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.845482 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca" pod "1a17ed26-e15b-483a-ac57-154345b63758" (UID: "1a17ed26-e15b-483a-ac57-154345b63758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:15:54.846656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.846374 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr" pod "1a17ed26-e15b-483a-ac57-154345b63758" (UID: "1a17ed26-e15b-483a-ac57-154345b63758"). InnerVolumeSpecName "kube-api-access-d7dkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:54.946088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.945134 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:54.946088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.945377 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dkr\" (UniqueName: \"kubernetes.io/projected/1a17ed26-e15b-483a-ac57-154345b63758-kube-api-access-d7dkr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:54.946088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:54.945475 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a17ed26-e15b-483a-ac57-154345b63758-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:55.164043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.150309 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk"] Nov 03 03:15:55.235766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.235739 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" event={"ID":"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7","Type":"ContainerStarted","Data":"cb081bdd83535315fbfdb8c89f729f3367402ab3d425fa51a4934bc516278ebf"} Nov 03 03:15:55.237286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.237261 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" Nov 03 03:15:55.237286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.237263 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7" event={"ID":"1a17ed26-e15b-483a-ac57-154345b63758","Type":"ContainerDied","Data":"7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5"} Nov 03 03:15:55.237448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.237321 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c209aa62ef920e60854027f574c1bcce2d3c8b91b2320c0348419f8b5e131f5" Nov 03 03:15:55.484927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.481518 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:55.484927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.482380 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:55.484927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:55.483264 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:15:56.247113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:56.247081 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" event={"ID":"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7","Type":"ContainerStarted","Data":"e5cb8a46da01534f0820d92b9bce1ca7da28d139620922af8bef5f81f79efbee"} Nov 03 03:15:56.365373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:56.365309 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" podStartSLOduration=2.365295936 podStartE2EDuration="2.365295936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:56.362352437 +0000 UTC m=+847.515040858" watchObservedRunningTime="2026-11-03 03:15:56.365295936 +0000 UTC m=+847.517984412" Nov 03 03:15:57.255427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:57.255390 5275 generic.go:356] "Generic (PLEG): container finished" podID="cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" containerID="e5cb8a46da01534f0820d92b9bce1ca7da28d139620922af8bef5f81f79efbee" exitCode=0 Nov 03 03:15:57.255989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:57.255457 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" event={"ID":"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7","Type":"ContainerDied","Data":"e5cb8a46da01534f0820d92b9bce1ca7da28d139620922af8bef5f81f79efbee"} Nov 03 03:15:58.262787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.262749 5275 generic.go:356] "Generic (PLEG): container finished" podID="13791e77-2f7c-432a-8845-155113624cc3" containerID="0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c" exitCode=0 Nov 03 03:15:58.263310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.262824 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerDied","Data":"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c"} Nov 03 03:15:58.263650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.263330 5275 scope.go:122] "RemoveContainer" containerID="0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c" Nov 03 03:15:58.264696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.264665 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:58.264836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.264758 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:58.264836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.264792 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:15:58.406412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.406347 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:58.485686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.485655 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\" (UniqueName: \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\") pod \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " Nov 03 03:15:58.485898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.485747 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\") pod \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " Nov 03 03:15:58.485898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.485825 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\") pod \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\" (UID: \"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7\") " Nov 03 03:15:58.486422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.486391 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca" pod "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" (UID: "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:58.488385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.488347 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj" pod "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" (UID: "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7"). InnerVolumeSpecName "kube-api-access-ph8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:58.488644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.488624 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca" pod "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" (UID: "cfdb20b9-628c-46cc-b4fa-10f110a8c5d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:15:58.587364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.587326 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:58.587364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.587358 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8nj\" (UniqueName: \"kubernetes.io/projected/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-kube-api-access-ph8nj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:58.587364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:58.587368 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:15:59.282316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.282286 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" Nov 03 03:15:59.282977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.282286 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk" event={"ID":"cfdb20b9-628c-46cc-b4fa-10f110a8c5d7","Type":"ContainerDied","Data":"cb081bdd83535315fbfdb8c89f729f3367402ab3d425fa51a4934bc516278ebf"} Nov 03 03:15:59.283094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.283050 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb081bdd83535315fbfdb8c89f729f3367402ab3d425fa51a4934bc516278ebf" Nov 03 03:15:59.285762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.285727 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerStarted","Data":"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752"} Nov 03 03:15:59.571931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.571282 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:15:59.571931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.571417 5275 patch_prober.go:29] interesting pod/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 03:15:59.571931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:15:59.571465 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 03:16:00.565302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:00.565265 5275 patch_prober.go:29] interesting pod/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 03:16:00.565942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:00.565911 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 03:16:01.563328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:01.563292 5275 patch_prober.go:29] interesting pod/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 03:16:01.563634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:01.563604 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 03:16:02.273961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.273911 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx"] Nov 03 03:16:02.274669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274644 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" containerName="revoke-keys" Nov 03 03:16:02.274752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274674 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" containerName="revoke-keys" Nov 03 03:16:02.274752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274713 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1a17ed26-e15b-483a-ac57-154345b63758" containerName="revoke-keys" Nov 03 03:16:02.274752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274721 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a17ed26-e15b-483a-ac57-154345b63758" containerName="revoke-keys" Nov 03 03:16:02.274961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274836 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" containerName="revoke-keys" Nov 03 03:16:02.274961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.274892 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a17ed26-e15b-483a-ac57-154345b63758" containerName="revoke-keys" Nov 03 03:16:02.277691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.277670 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.329263 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:02.329208 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:16:02.332777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.332752 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:16:02.382455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.382412 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:16:02.382455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.382428 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:16:02.382756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.382512 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:16:02.390161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.386912 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx"] Nov 03 03:16:02.428946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.428903 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.429178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.429027 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.429178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.429081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z4w\" (UniqueName: \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.530036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.529952 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.530036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.530015 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z4w\" (UniqueName: \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.530241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.530090 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.530639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.530616 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.532537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.532517 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.572884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.571325 5275 patch_prober.go:29] interesting pod/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 03:16:02.572884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.571402 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 03:16:02.592006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.591975 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z4w\" (UniqueName: \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\") pod \"maas-api-revoke-keys-e2e-ait-56094c67-t2nsx\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:02.889948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:02.889840 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:03.277710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:03.277676 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx"] Nov 03 03:16:03.279953 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:03.279921 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017d42c1_1a69_41a5_93a7_07b4924ef2de.slice/crio-382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5 WatchSource:0}: Error finding container 382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5: Status 404 returned error can't find the container with id 382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5 Nov 03 03:16:03.309473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:03.309439 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" event={"ID":"017d42c1-1a69-41a5-93a7-07b4924ef2de","Type":"ContainerStarted","Data":"382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5"} Nov 03 03:16:03.533960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:03.533870 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:16:03.562382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:03.562259 5275 patch_prober.go:29] interesting pod/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 03:16:03.562382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:03.562319 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 03:16:04.317068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:04.317031 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" event={"ID":"017d42c1-1a69-41a5-93a7-07b4924ef2de","Type":"ContainerStarted","Data":"f6adc05149d7a7ddd16ce1ff8d39c55eba8413ad654f4c3f9f527e23d0d9951f"} Nov 03 03:16:04.435474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:04.435422 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" podStartSLOduration=2.435407631 podStartE2EDuration="2.435407631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:04.432351871 +0000 UTC m=+855.585040291" watchObservedRunningTime="2026-11-03 03:16:04.435407631 +0000 UTC m=+855.588096046" Nov 03 03:16:04.561002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:04.560974 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:16:04.561292 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:04.561268 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:16:04.561998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:04.561982 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:16:05.324829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:05.324799 5275 generic.go:356] "Generic (PLEG): container finished" podID="017d42c1-1a69-41a5-93a7-07b4924ef2de" containerID="f6adc05149d7a7ddd16ce1ff8d39c55eba8413ad654f4c3f9f527e23d0d9951f" exitCode=0 Nov 03 03:16:05.325329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:05.324895 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" event={"ID":"017d42c1-1a69-41a5-93a7-07b4924ef2de","Type":"ContainerDied","Data":"f6adc05149d7a7ddd16ce1ff8d39c55eba8413ad654f4c3f9f527e23d0d9951f"} Nov 03 03:16:06.465587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.465555 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:06.573592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.573561 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\") pod \"017d42c1-1a69-41a5-93a7-07b4924ef2de\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " Nov 03 03:16:06.573799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.573669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\") pod \"017d42c1-1a69-41a5-93a7-07b4924ef2de\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " Nov 03 03:16:06.573799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.573735 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\" (UniqueName: \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\") pod \"017d42c1-1a69-41a5-93a7-07b4924ef2de\" (UID: \"017d42c1-1a69-41a5-93a7-07b4924ef2de\") " Nov 03 03:16:06.574015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.573991 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca" pod "017d42c1-1a69-41a5-93a7-07b4924ef2de" (UID: "017d42c1-1a69-41a5-93a7-07b4924ef2de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:06.574135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.574120 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:06.575970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.575945 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca" pod "017d42c1-1a69-41a5-93a7-07b4924ef2de" (UID: "017d42c1-1a69-41a5-93a7-07b4924ef2de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:06.575970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.575952 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w" pod "017d42c1-1a69-41a5-93a7-07b4924ef2de" (UID: "017d42c1-1a69-41a5-93a7-07b4924ef2de"). InnerVolumeSpecName "kube-api-access-w2z4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:06.675531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.675491 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017d42c1-1a69-41a5-93a7-07b4924ef2de-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:06.675531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:06.675526 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2z4w\" (UniqueName: \"kubernetes.io/projected/017d42c1-1a69-41a5-93a7-07b4924ef2de-kube-api-access-w2z4w\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:07.338067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:07.338036 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" Nov 03 03:16:07.338067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:07.338051 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx" event={"ID":"017d42c1-1a69-41a5-93a7-07b4924ef2de","Type":"ContainerDied","Data":"382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5"} Nov 03 03:16:07.338282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:07.338103 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="382cbd7665d5924aef0d69d2c8afd5580395a710a0a2816a68a47c051e264ef5" Nov 03 03:16:08.534594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.534531 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" podUID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" containerName="payload-processing" containerID="cri-o://22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc" gracePeriod=25 Nov 03 03:16:08.795217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.795160 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:16:08.898536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.898505 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\" (UniqueName: \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\") pod \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " Nov 03 03:16:08.898681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.898560 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\") pod \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\" (UID: \"bd386991-cb34-4776-8b3b-ea2e7dd63b5e\") " Nov 03 03:16:08.898974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.898941 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume" pod "bd386991-cb34-4776-8b3b-ea2e7dd63b5e" (UID: "bd386991-cb34-4776-8b3b-ea2e7dd63b5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:08.900820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.900796 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2" pod "bd386991-cb34-4776-8b3b-ea2e7dd63b5e" (UID: "bd386991-cb34-4776-8b3b-ea2e7dd63b5e"). InnerVolumeSpecName "kube-api-access-zgks2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:08.999818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.999781 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgks2\" (UniqueName: \"kubernetes.io/projected/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-kube-api-access-zgks2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:08.999818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:08.999815 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd386991-cb34-4776-8b3b-ea2e7dd63b5e-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:09.350488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.350453 5275 generic.go:356] "Generic (PLEG): container finished" podID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" containerID="22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc" exitCode=0 Nov 03 03:16:09.350669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.350506 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" Nov 03 03:16:09.350669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.350533 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" event={"ID":"bd386991-cb34-4776-8b3b-ea2e7dd63b5e","Type":"ContainerDied","Data":"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc"} Nov 03 03:16:09.350669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.350582 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm" event={"ID":"bd386991-cb34-4776-8b3b-ea2e7dd63b5e","Type":"ContainerDied","Data":"f1073ff11b018fd684f67d401744f9c327c7438a5afd3769c740ac0124cb6e6c"} Nov 03 03:16:09.350669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.350604 5275 scope.go:122] "RemoveContainer" containerID="22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc" Nov 03 03:16:09.360055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.360036 5275 scope.go:122] "RemoveContainer" containerID="22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc" Nov 03 03:16:09.360330 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:09.360310 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc\": container with ID starting with 22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc not found: ID does not exist" containerID="22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc" Nov 03 03:16:09.360373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.360339 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc"} err="failed to get container status \"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc\": rpc error: code = NotFound desc = could not find container \"22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc\": container with ID starting with 22be434480c28abbfb0806fa215688975ea4474410b334f5c290cb7896dbd5bc not found: ID does not exist" Nov 03 03:16:09.516255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.516223 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:16:09.565607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:09.565578 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b4ff4779-577bdcb8d5-h5ljm"] Nov 03 03:16:10.669953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.669918 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl"] Nov 03 03:16:10.670376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670350 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" containerName="payload-processing" Nov 03 03:16:10.670376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670362 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" containerName="payload-processing" Nov 03 03:16:10.670459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670395 5275 cpu_manager.go:414] "containerMap: removing container" podUID="017d42c1-1a69-41a5-93a7-07b4924ef2de" containerName="revoke-keys" Nov 03 03:16:10.670459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670400 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017d42c1-1a69-41a5-93a7-07b4924ef2de" containerName="revoke-keys" Nov 03 03:16:10.670539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670462 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="017d42c1-1a69-41a5-93a7-07b4924ef2de" containerName="revoke-keys" Nov 03 03:16:10.670539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.670473 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" containerName="payload-processing" Nov 03 03:16:10.673208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.673191 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.725249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.725215 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:16:10.725436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.725397 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:16:10.725436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.725397 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:16:10.775041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.775009 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:16:10.777607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.777587 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl"] Nov 03 03:16:10.818173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.818141 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.818307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.818288 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fj7\" (UniqueName: \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.818365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.818347 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.919213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.919174 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fj7\" (UniqueName: \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.919383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.919221 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.919383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.919290 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.920237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.920094 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.921749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.921727 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:10.994269 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:10.994234 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd386991-cb34-4776-8b3b-ea2e7dd63b5e" path="/var/lib/kubelet/pods/bd386991-cb34-4776-8b3b-ea2e7dd63b5e/volumes" Nov 03 03:16:11.011597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:11.011530 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fj7\" (UniqueName: \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\") pod \"maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:11.284384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:11.284351 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:11.663821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:11.663784 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl"] Nov 03 03:16:11.665500 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:11.665474 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d9fa89_b509_4595_8e0d_14900c01ce84.slice/crio-6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2 WatchSource:0}: Error finding container 6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2: Status 404 returned error can't find the container with id 6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2 Nov 03 03:16:12.371756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:12.371718 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" event={"ID":"29d9fa89-b509-4595-8e0d-14900c01ce84","Type":"ContainerStarted","Data":"56ac9c337bed0d8c01472831ad92b5ced37736b7328b055fc2dfd8079ac54dd6"} Nov 03 03:16:12.372195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:12.371773 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" event={"ID":"29d9fa89-b509-4595-8e0d-14900c01ce84","Type":"ContainerStarted","Data":"6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2"} Nov 03 03:16:12.489265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:12.489208 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" podStartSLOduration=2.489192947 podStartE2EDuration="2.489192947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:12.48862258 +0000 UTC m=+863.641311003" watchObservedRunningTime="2026-11-03 03:16:12.489192947 +0000 UTC m=+863.641881367" Nov 03 03:16:13.377957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.377923 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7"] Nov 03 03:16:13.380307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.380282 5275 generic.go:356] "Generic (PLEG): container finished" podID="29d9fa89-b509-4595-8e0d-14900c01ce84" containerID="56ac9c337bed0d8c01472831ad92b5ced37736b7328b055fc2dfd8079ac54dd6" exitCode=0 Nov 03 03:16:13.380457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.380357 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" event={"ID":"29d9fa89-b509-4595-8e0d-14900c01ce84","Type":"ContainerDied","Data":"56ac9c337bed0d8c01472831ad92b5ced37736b7328b055fc2dfd8079ac54dd6"} Nov 03 03:16:13.428145 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.428108 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4ff4779-sqsz7"] Nov 03 03:16:13.790486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.790454 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm"] Nov 03 03:16:13.793063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.793048 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:13.853464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.853432 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm"] Nov 03 03:16:13.951502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.951461 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:13.951679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.951613 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:13.951679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:13.951650 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk9s\" (UniqueName: \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.052533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.052455 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.052674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.052541 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.052674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.052567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk9s\" (UniqueName: \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.053222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.053196 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.054798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.054780 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.113332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.113300 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk9s\" (UniqueName: \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\") pod \"maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.404918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.404890 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:14.554883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.554853 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:14.657172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.657090 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\" (UniqueName: \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\") pod \"29d9fa89-b509-4595-8e0d-14900c01ce84\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " Nov 03 03:16:14.657172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.657157 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\") pod \"29d9fa89-b509-4595-8e0d-14900c01ce84\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " Nov 03 03:16:14.657400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.657292 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\") pod \"29d9fa89-b509-4595-8e0d-14900c01ce84\" (UID: \"29d9fa89-b509-4595-8e0d-14900c01ce84\") " Nov 03 03:16:14.657533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.657503 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca" pod "29d9fa89-b509-4595-8e0d-14900c01ce84" (UID: "29d9fa89-b509-4595-8e0d-14900c01ce84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:14.657629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.657617 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:14.659381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.659349 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca" pod "29d9fa89-b509-4595-8e0d-14900c01ce84" (UID: "29d9fa89-b509-4595-8e0d-14900c01ce84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:14.659500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.659387 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7" pod "29d9fa89-b509-4595-8e0d-14900c01ce84" (UID: "29d9fa89-b509-4595-8e0d-14900c01ce84"). InnerVolumeSpecName "kube-api-access-h7fj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:14.758370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.758336 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d9fa89-b509-4595-8e0d-14900c01ce84-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:14.758370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.758366 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fj7\" (UniqueName: \"kubernetes.io/projected/29d9fa89-b509-4595-8e0d-14900c01ce84-kube-api-access-h7fj7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:14.801670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.801644 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm"] Nov 03 03:16:14.803701 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:14.803672 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f045813_a5fa_4818_9a43_b76a3d2a9377.slice/crio-3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341 WatchSource:0}: Error finding container 3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341: Status 404 returned error can't find the container with id 3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341 Nov 03 03:16:14.879328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.879296 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:16:14.879768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.879756 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29d9fa89-b509-4595-8e0d-14900c01ce84" containerName="revoke-keys" Nov 03 03:16:14.879834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.879770 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d9fa89-b509-4595-8e0d-14900c01ce84" containerName="revoke-keys" Nov 03 03:16:14.879906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.879884 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d9fa89-b509-4595-8e0d-14900c01ce84" containerName="revoke-keys" Nov 03 03:16:14.882798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.882777 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.941592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.941053 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\"" Nov 03 03:16:14.946024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.945412 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:16:14.961348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961244 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.961821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.961821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961414 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.961821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961543 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.961821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961575 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcfp\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.961821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961703 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.962290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961923 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.962290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.961958 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.962290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.962091 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:14.968180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.968156 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk"] Nov 03 03:16:14.986368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.986341 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a17ed26-e15b-483a-ac57-154345b63758" path="/var/lib/kubelet/pods/1a17ed26-e15b-483a-ac57-154345b63758/volumes" Nov 03 03:16:14.991333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:14.991306 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4fb905b5-p9rnk"] Nov 03 03:16:15.044146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.044111 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:16:15.044411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.044384 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" containerID="cri-o://2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995" gracePeriod=30 Nov 03 03:16:15.063035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063001 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063086 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063166 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063196 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063276 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063255 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063306 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcfp\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063342 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063416 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.063549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063440 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.064202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.063983 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.064202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.064125 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.064202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.064167 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.064428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.064283 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.064666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.064642 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.066216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.066194 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.066681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.066657 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.128027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.127994 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.128171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.128157 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcfp\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.195705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.195589 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:15.394544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.394520 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" Nov 03 03:16:15.394743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.394551 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl" event={"ID":"29d9fa89-b509-4595-8e0d-14900c01ce84","Type":"ContainerDied","Data":"6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2"} Nov 03 03:16:15.394743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.394607 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5a9b5449189066662d179ef5abb9ee9cb27723002294669c14cc03e1e0c1d2" Nov 03 03:16:15.396368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.396335 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" event={"ID":"3f045813-a5fa-4818-9a43-b76a3d2a9377","Type":"ContainerStarted","Data":"0bcc8d8a5d42250e527fbef907bbc188cc1274d4952faf592fe3f9d696c13558"} Nov 03 03:16:15.396504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.396382 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" event={"ID":"3f045813-a5fa-4818-9a43-b76a3d2a9377","Type":"ContainerStarted","Data":"3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341"} Nov 03 03:16:15.515024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.514915 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" podStartSLOduration=2.514898263 podStartE2EDuration="2.514898263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:15.512168303 +0000 UTC m=+866.664856739" watchObservedRunningTime="2026-11-03 03:16:15.514898263 +0000 UTC m=+866.667586690" Nov 03 03:16:15.625446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.625419 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:16:15.627310 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:15.627285 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12a6b3d_1e9d_403f_9108_519c4b4bc898.slice/crio-e74f48f322f2de7dc7c9381e8606aa4d108d96c0108e0a1bb9ebf0c467fb8e39 WatchSource:0}: Error finding container e74f48f322f2de7dc7c9381e8606aa4d108d96c0108e0a1bb9ebf0c467fb8e39: Status 404 returned error can't find the container with id e74f48f322f2de7dc7c9381e8606aa4d108d96c0108e0a1bb9ebf0c467fb8e39 Nov 03 03:16:15.629676 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.629640 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:15.629772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.629725 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:15.629835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:15.629771 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:16.036121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.036086 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:16:16.405596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.405503 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerStarted","Data":"52abbf6dfc05e17e71c48a48f49caf88ebe2c3df56285793cdf1fccb5904ed23"} Nov 03 03:16:16.405596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.405562 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerStarted","Data":"e74f48f322f2de7dc7c9381e8606aa4d108d96c0108e0a1bb9ebf0c467fb8e39"} Nov 03 03:16:16.407198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.407173 5275 generic.go:356] "Generic (PLEG): container finished" podID="3f045813-a5fa-4818-9a43-b76a3d2a9377" containerID="0bcc8d8a5d42250e527fbef907bbc188cc1274d4952faf592fe3f9d696c13558" exitCode=0 Nov 03 03:16:16.407324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.407260 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" event={"ID":"3f045813-a5fa-4818-9a43-b76a3d2a9377","Type":"ContainerDied","Data":"0bcc8d8a5d42250e527fbef907bbc188cc1274d4952faf592fe3f9d696c13558"} Nov 03 03:16:16.438083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.438053 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:16:16.441272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.441254 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.502593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.502560 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-68b670-openshift-default-dockercfg-tnxvt\"" Nov 03 03:16:16.507108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.507073 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:16:16.531952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.531887 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podStartSLOduration=2.531867208 podStartE2EDuration="2.531867208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:16.529271175 +0000 UTC m=+867.681959633" watchObservedRunningTime="2026-11-03 03:16:16.531867208 +0000 UTC m=+867.684555630" Nov 03 03:16:16.578327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578330 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578357 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578496 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578526 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578659 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578753 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwf8b\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578784 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.578939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.578923 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682455 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwf8b\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682508 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682548 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682603 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682635 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682673 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682719 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682750 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.682863 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.690365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.683744 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.691083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.690488 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.691083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.690729 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.691083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.690986 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.691548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.691519 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.698219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.698154 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.699000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.698834 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.745302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.745271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwf8b\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.750208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.750183 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.756514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.756496 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:16.983692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:16.983610 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfdb20b9-628c-46cc-b4fa-10f110a8c5d7" path="/var/lib/kubelet/pods/cfdb20b9-628c-46cc-b4fa-10f110a8c5d7/volumes" Nov 03 03:16:17.186179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.186155 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:16:17.188006 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:17.187976 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f0a28b_af55_4874_a30f_c4452d75f2a5.slice/crio-16d6efbbd6302a745d83afc9839929e23dc69c2c23e65a51734d3800745c2e04 WatchSource:0}: Error finding container 16d6efbbd6302a745d83afc9839929e23dc69c2c23e65a51734d3800745c2e04: Status 404 returned error can't find the container with id 16d6efbbd6302a745d83afc9839929e23dc69c2c23e65a51734d3800745c2e04 Nov 03 03:16:17.190364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.190331 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:17.190428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.190395 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:17.190472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.190426 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:17.196465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.196443 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:17.201773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.201750 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:17.415928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.415892 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerStarted","Data":"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0"} Nov 03 03:16:17.416121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.415955 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerStarted","Data":"16d6efbbd6302a745d83afc9839929e23dc69c2c23e65a51734d3800745c2e04"} Nov 03 03:16:17.416340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.416310 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:17.417344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.417319 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:17.630805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.630767 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:17.667003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.666952 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podStartSLOduration=1.666934573 podStartE2EDuration="1.666934573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:17.543518006 +0000 UTC m=+868.696206430" watchObservedRunningTime="2026-11-03 03:16:17.666934573 +0000 UTC m=+868.819623026" Nov 03 03:16:17.699417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.699380 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\") pod \"3f045813-a5fa-4818-9a43-b76a3d2a9377\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " Nov 03 03:16:17.699566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.699462 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\" (UniqueName: \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\") pod \"3f045813-a5fa-4818-9a43-b76a3d2a9377\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " Nov 03 03:16:17.699603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.699573 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\") pod \"3f045813-a5fa-4818-9a43-b76a3d2a9377\" (UID: \"3f045813-a5fa-4818-9a43-b76a3d2a9377\") " Nov 03 03:16:17.700183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.700151 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca" pod "3f045813-a5fa-4818-9a43-b76a3d2a9377" (UID: "3f045813-a5fa-4818-9a43-b76a3d2a9377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:17.702056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.702031 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s" pod "3f045813-a5fa-4818-9a43-b76a3d2a9377" (UID: "3f045813-a5fa-4818-9a43-b76a3d2a9377"). InnerVolumeSpecName "kube-api-access-xgk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:17.702221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.702201 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca" pod "3f045813-a5fa-4818-9a43-b76a3d2a9377" (UID: "3f045813-a5fa-4818-9a43-b76a3d2a9377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:17.757737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.757624 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:17.822081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.821535 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.822081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.821567 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f045813-a5fa-4818-9a43-b76a3d2a9377-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:17.822081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:17.821579 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgk9s\" (UniqueName: \"kubernetes.io/projected/3f045813-a5fa-4818-9a43-b76a3d2a9377-kube-api-access-xgk9s\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:18.424008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:18.423926 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" Nov 03 03:16:18.424008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:18.423951 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm" event={"ID":"3f045813-a5fa-4818-9a43-b76a3d2a9377","Type":"ContainerDied","Data":"3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341"} Nov 03 03:16:18.424008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:18.423996 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b800328048ddebabf0368d70ce54588279c8d5a19cc45a4afa36849c997d341" Nov 03 03:16:18.761895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:18.761799 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:19.429768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:19.429727 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:19.430524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:19.430504 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:16:20.301120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.301095 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:16:20.439200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.439112 5275 generic.go:356] "Generic (PLEG): container finished" podID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerID="2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995" exitCode=0 Nov 03 03:16:20.439344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.439189 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerDied","Data":"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995"} Nov 03 03:16:20.439344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.439222 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" Nov 03 03:16:20.439344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.439252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j" event={"ID":"faf5ad3d-e8d1-46bb-9f65-769b87057ccf","Type":"ContainerDied","Data":"d6602c482075ad0ff9a4033689044ffdbb9c19d06a7579f45f42bf65bdc19819"} Nov 03 03:16:20.439344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.439276 5275 scope.go:122] "RemoveContainer" containerID="2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995" Nov 03 03:16:20.447398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447375 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447414 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447535 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447586 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447621 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447661 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447758 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447809 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.447987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.447874 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\") pod \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\" (UID: \"faf5ad3d-e8d1-46bb-9f65-769b87057ccf\") " Nov 03 03:16:20.449128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.448500 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:20.449128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.448628 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:20.449128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.448646 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:20.449128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.448794 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:20.449128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.449100 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:20.450270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.450244 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:20.450355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.450283 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:20.450831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.450807 5275 scope.go:122] "RemoveContainer" containerID="298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4" Nov 03 03:16:20.451094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.451064 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:16:20.451332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.451309 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc" pod "faf5ad3d-e8d1-46bb-9f65-769b87057ccf" (UID: "faf5ad3d-e8d1-46bb-9f65-769b87057ccf"). InnerVolumeSpecName "kube-api-access-fh5gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:20.471022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.470994 5275 scope.go:122] "RemoveContainer" containerID="2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995" Nov 03 03:16:20.471357 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:20.471331 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995\": container with ID starting with 2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995 not found: ID does not exist" containerID="2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995" Nov 03 03:16:20.471422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.471365 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995"} err="failed to get container status \"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995\": rpc error: code = NotFound desc = could not find container \"2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995\": container with ID starting with 2f577a840a0038d52cf24cd2cb073e8711e7f5cae88dc6441505cc46b995b995 not found: ID does not exist" Nov 03 03:16:20.471422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.471386 5275 scope.go:122] "RemoveContainer" containerID="298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4" Nov 03 03:16:20.471693 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:20.471677 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4\": container with ID starting with 298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4 not found: ID does not exist" containerID="298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4" Nov 03 03:16:20.471730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.471703 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4"} err="failed to get container status \"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4\": rpc error: code = NotFound desc = could not find container \"298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4\": container with ID starting with 298ee6dfd2b304d6ca34f69697fe3d362831a26c099fbe5ef3ca74c64fc09bd4 not found: ID does not exist" Nov 03 03:16:20.549169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549121 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549157 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549171 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549184 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549201 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549213 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549227 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5gc\" (UniqueName: \"kubernetes.io/projected/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-kube-api-access-fh5gc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549241 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.549465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.549253 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf5ad3d-e8d1-46bb-9f65-769b87057ccf-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:20.865371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.865334 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:16:20.914751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.914716 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4fb905b5-gw-openshift-default-54b69f7769-9b46j"] Nov 03 03:16:20.984254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:20.984224 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" path="/var/lib/kubelet/pods/faf5ad3d-e8d1-46bb-9f65-769b87057ccf/volumes" Nov 03 03:16:21.036753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.036710 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" podUID="49341fdc-5e87-4947-9847-0f86708eb515" containerName="payload-processing" containerID="cri-o://c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61" gracePeriod=25 Nov 03 03:16:21.290577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.290514 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:16:21.447436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.447405 5275 generic.go:356] "Generic (PLEG): container finished" podID="49341fdc-5e87-4947-9847-0f86708eb515" containerID="c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61" exitCode=0 Nov 03 03:16:21.447861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.447451 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" Nov 03 03:16:21.447861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.447484 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" event={"ID":"49341fdc-5e87-4947-9847-0f86708eb515","Type":"ContainerDied","Data":"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61"} Nov 03 03:16:21.447861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.447538 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng" event={"ID":"49341fdc-5e87-4947-9847-0f86708eb515","Type":"ContainerDied","Data":"179b6dc45d0a094a5ee57b1e164691a01da145b19b73937819891fab02f1958d"} Nov 03 03:16:21.447861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.447563 5275 scope.go:122] "RemoveContainer" containerID="c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61" Nov 03 03:16:21.457364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.457346 5275 scope.go:122] "RemoveContainer" containerID="c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61" Nov 03 03:16:21.457693 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:21.457674 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61\": container with ID starting with c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61 not found: ID does not exist" containerID="c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61" Nov 03 03:16:21.457734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.457704 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61"} err="failed to get container status \"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61\": rpc error: code = NotFound desc = could not find container \"c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61\": container with ID starting with c315f3c70b97086b3d43478d3491445e52f371a4173a577d959eae17a2c50d61 not found: ID does not exist" Nov 03 03:16:21.458569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.458556 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\") pod \"49341fdc-5e87-4947-9847-0f86708eb515\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " Nov 03 03:16:21.458669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.458653 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\" (UniqueName: \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\") pod \"49341fdc-5e87-4947-9847-0f86708eb515\" (UID: \"49341fdc-5e87-4947-9847-0f86708eb515\") " Nov 03 03:16:21.459002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.458975 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume" pod "49341fdc-5e87-4947-9847-0f86708eb515" (UID: "49341fdc-5e87-4947-9847-0f86708eb515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:21.463090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.463057 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h" pod "49341fdc-5e87-4947-9847-0f86708eb515" (UID: "49341fdc-5e87-4947-9847-0f86708eb515"). InnerVolumeSpecName "kube-api-access-pk67h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:21.560068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.559966 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49341fdc-5e87-4947-9847-0f86708eb515-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:21.560068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.560004 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk67h\" (UniqueName: \"kubernetes.io/projected/49341fdc-5e87-4947-9847-0f86708eb515-kube-api-access-pk67h\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:21.868417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.868334 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:16:21.918126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:21.918094 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56094c67-67598699f9-v84ng"] Nov 03 03:16:22.333569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:22.333540 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:16:22.455641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:22.455611 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:16:22.983313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:22.983279 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49341fdc-5e87-4947-9847-0f86708eb515" path="/var/lib/kubelet/pods/49341fdc-5e87-4947-9847-0f86708eb515/volumes" Nov 03 03:16:26.596457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.596427 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 03:16:26.597108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597081 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3f045813-a5fa-4818-9a43-b76a3d2a9377" containerName="revoke-keys" Nov 03 03:16:26.597108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597107 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f045813-a5fa-4818-9a43-b76a3d2a9377" containerName="revoke-keys" Nov 03 03:16:26.597270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597148 5275 cpu_manager.go:414] "containerMap: removing container" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:26.597270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597156 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:26.597270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597183 5275 cpu_manager.go:414] "containerMap: removing container" podUID="49341fdc-5e87-4947-9847-0f86708eb515" containerName="payload-processing" Nov 03 03:16:26.597270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597191 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49341fdc-5e87-4947-9847-0f86708eb515" containerName="payload-processing" Nov 03 03:16:26.597392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597274 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f045813-a5fa-4818-9a43-b76a3d2a9377" containerName="revoke-keys" Nov 03 03:16:26.597392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597285 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:26.597392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597292 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:26.597392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.597299 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="49341fdc-5e87-4947-9847-0f86708eb515" containerName="payload-processing" Nov 03 03:16:26.599297 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.599282 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.617834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.617809 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.618000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.617925 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnst\" (UniqueName: \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.725092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.725034 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-de27f5\"" Nov 03 03:16:26.726284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.725295 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-de27f5-dockercfg-k494x\"" Nov 03 03:16:26.726284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.725430 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.726284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.725525 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnst\" (UniqueName: \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.726284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.726224 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 03:16:26.726547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.726430 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.788112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.788082 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnst\" (UniqueName: \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\") pod \"payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:26.911837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:26.911751 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:27.299649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.299619 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 03:16:27.301382 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:27.301357 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc744aa1_fa60_4f2a_9b1e_e625bfd7a9e0.slice/crio-145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503 WatchSource:0}: Error finding container 145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503: Status 404 returned error can't find the container with id 145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503 Nov 03 03:16:27.334073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.334038 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" podUID="6653ef49-967b-41e6-80b1-0f60385c74df" containerName="payload-processing" containerID="cri-o://87a77a4e7e889ae122eda676335f6b808022975679fe531b6a70ba6a71685472" gracePeriod=25 Nov 03 03:16:27.423481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.421643 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:16:27.456761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.456726 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" podUID="9f235e68-b3f9-4603-be50-fef9e0ab6366" containerName="payload-pre-processing" containerID="cri-o://b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315" gracePeriod=25 Nov 03 03:16:27.493809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.493780 5275 generic.go:356] "Generic (PLEG): container finished" podID="6653ef49-967b-41e6-80b1-0f60385c74df" containerID="87a77a4e7e889ae122eda676335f6b808022975679fe531b6a70ba6a71685472" exitCode=0 Nov 03 03:16:27.494007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.493890 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" event={"ID":"6653ef49-967b-41e6-80b1-0f60385c74df","Type":"ContainerDied","Data":"87a77a4e7e889ae122eda676335f6b808022975679fe531b6a70ba6a71685472"} Nov 03 03:16:27.495897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.495869 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" event={"ID":"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0","Type":"ContainerStarted","Data":"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1"} Nov 03 03:16:27.496033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.495920 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" event={"ID":"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0","Type":"ContainerStarted","Data":"145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503"} Nov 03 03:16:27.496033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.495980 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:27.595570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.595541 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:16:27.635841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.635810 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\" (UniqueName: \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\") pod \"6653ef49-967b-41e6-80b1-0f60385c74df\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " Nov 03 03:16:27.636258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.635906 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\") pod \"6653ef49-967b-41e6-80b1-0f60385c74df\" (UID: \"6653ef49-967b-41e6-80b1-0f60385c74df\") " Nov 03 03:16:27.636338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.636310 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume" pod "6653ef49-967b-41e6-80b1-0f60385c74df" (UID: "6653ef49-967b-41e6-80b1-0f60385c74df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:27.639057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.638990 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7" pod "6653ef49-967b-41e6-80b1-0f60385c74df" (UID: "6653ef49-967b-41e6-80b1-0f60385c74df"). InnerVolumeSpecName "kube-api-access-ftlc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:27.663490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.663426 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" podStartSLOduration=1.663406507 podStartE2EDuration="1.663406507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:27.660354084 +0000 UTC m=+878.813042518" watchObservedRunningTime="2026-11-03 03:16:27.663406507 +0000 UTC m=+878.816094932" Nov 03 03:16:27.720099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.720074 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:16:27.736991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.736957 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\" (UniqueName: \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\") pod \"9f235e68-b3f9-4603-be50-fef9e0ab6366\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " Nov 03 03:16:27.737143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.737007 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\") pod \"9f235e68-b3f9-4603-be50-fef9e0ab6366\" (UID: \"9f235e68-b3f9-4603-be50-fef9e0ab6366\") " Nov 03 03:16:27.737364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.737347 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlc7\" (UniqueName: \"kubernetes.io/projected/6653ef49-967b-41e6-80b1-0f60385c74df-kube-api-access-ftlc7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:27.737440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.737370 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6653ef49-967b-41e6-80b1-0f60385c74df-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:27.737440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.737408 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume" pod "9f235e68-b3f9-4603-be50-fef9e0ab6366" (UID: "9f235e68-b3f9-4603-be50-fef9e0ab6366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:27.739570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.739540 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss" pod "9f235e68-b3f9-4603-be50-fef9e0ab6366" (UID: "9f235e68-b3f9-4603-be50-fef9e0ab6366"). InnerVolumeSpecName "kube-api-access-4zkss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:27.840373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.840271 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkss\" (UniqueName: \"kubernetes.io/projected/9f235e68-b3f9-4603-be50-fef9e0ab6366-kube-api-access-4zkss\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:27.840373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:27.840312 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f235e68-b3f9-4603-be50-fef9e0ab6366-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:28.503058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.503022 5275 generic.go:356] "Generic (PLEG): container finished" podID="9f235e68-b3f9-4603-be50-fef9e0ab6366" containerID="b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315" exitCode=0 Nov 03 03:16:28.503254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.503072 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" Nov 03 03:16:28.503254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.503101 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" event={"ID":"9f235e68-b3f9-4603-be50-fef9e0ab6366","Type":"ContainerDied","Data":"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315"} Nov 03 03:16:28.503254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.503153 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp" event={"ID":"9f235e68-b3f9-4603-be50-fef9e0ab6366","Type":"ContainerDied","Data":"2646c3eba11d9fb8fc80adef9aeb7cd4647e820b0a05d8615cd3d62e149361fd"} Nov 03 03:16:28.503254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.503178 5275 scope.go:122] "RemoveContainer" containerID="b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315" Nov 03 03:16:28.504674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.504637 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" event={"ID":"6653ef49-967b-41e6-80b1-0f60385c74df","Type":"ContainerDied","Data":"edea2372867e0262c057b43c9f121af79ce9e2b8378632348bce969ed8cdcb19"} Nov 03 03:16:28.504789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.504660 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7" Nov 03 03:16:28.519332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.519314 5275 scope.go:122] "RemoveContainer" containerID="b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315" Nov 03 03:16:28.519647 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:28.519629 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315\": container with ID starting with b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315 not found: ID does not exist" containerID="b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315" Nov 03 03:16:28.519716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.519660 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315"} err="failed to get container status \"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315\": rpc error: code = NotFound desc = could not find container \"b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315\": container with ID starting with b599405b26bad76c540ddac39b1e1209b6121093537bee75bc75e3b684114315 not found: ID does not exist" Nov 03 03:16:28.519716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.519684 5275 scope.go:122] "RemoveContainer" containerID="87a77a4e7e889ae122eda676335f6b808022975679fe531b6a70ba6a71685472" Nov 03 03:16:28.679964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.679924 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:16:28.729623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.729595 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e21b4958-5c45bbcccf-n7pfp"] Nov 03 03:16:28.797636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.797600 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:16:28.801138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.801111 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e21b4958-59968cfbf9-n95d7"] Nov 03 03:16:28.984380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.984348 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6653ef49-967b-41e6-80b1-0f60385c74df" path="/var/lib/kubelet/pods/6653ef49-967b-41e6-80b1-0f60385c74df/volumes" Nov 03 03:16:28.984721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:28.984709 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f235e68-b3f9-4603-be50-fef9e0ab6366" path="/var/lib/kubelet/pods/9f235e68-b3f9-4603-be50-fef9e0ab6366/volumes" Nov 03 03:16:31.733943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.733903 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 03:16:31.734575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734555 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9f235e68-b3f9-4603-be50-fef9e0ab6366" containerName="payload-pre-processing" Nov 03 03:16:31.734670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734577 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f235e68-b3f9-4603-be50-fef9e0ab6366" containerName="payload-pre-processing" Nov 03 03:16:31.734670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734595 5275 cpu_manager.go:414] "containerMap: removing container" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:31.734670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734602 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf5ad3d-e8d1-46bb-9f65-769b87057ccf" containerName="istio-proxy" Nov 03 03:16:31.734670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734624 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6653ef49-967b-41e6-80b1-0f60385c74df" containerName="payload-processing" Nov 03 03:16:31.734670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734631 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6653ef49-967b-41e6-80b1-0f60385c74df" containerName="payload-processing" Nov 03 03:16:31.734964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734763 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f235e68-b3f9-4603-be50-fef9e0ab6366" containerName="payload-pre-processing" Nov 03 03:16:31.734964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.734778 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6653ef49-967b-41e6-80b1-0f60385c74df" containerName="payload-processing" Nov 03 03:16:31.737840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.737819 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:31.790102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.790071 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 03:16:31.837330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.837299 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-68b670\"" Nov 03 03:16:31.838234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.838209 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-68b670-dockercfg-tnhxv\"" Nov 03 03:16:31.884401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.884366 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:31.884559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.884428 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmbz\" (UniqueName: \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:31.985955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.985875 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmbz\" (UniqueName: \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:31.986103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.986010 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:31.986665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:31.986649 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:32.047560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.047524 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmbz\" (UniqueName: \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\") pod \"payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:32.050189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.050161 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:32.332507 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:32.332475 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:16:32.424079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.424022 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" podUID="caf42cf6-7d10-493c-91c6-3c2c1960c805" containerName="payload-pre-processing" containerID="cri-o://a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff" gracePeriod=25 Nov 03 03:16:32.425990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.425953 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 03:16:32.426933 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:16:32.426905 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29580c82_d705_4e84_a050_69f3610bec2c.slice/crio-cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531 WatchSource:0}: Error finding container cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531: Status 404 returned error can't find the container with id cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531 Nov 03 03:16:32.556125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.556092 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" event={"ID":"29580c82-d705-4e84-a050-69f3610bec2c","Type":"ContainerStarted","Data":"cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531"} Nov 03 03:16:32.671996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.671972 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:16:32.793575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.793538 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\" (UniqueName: \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\") pod \"caf42cf6-7d10-493c-91c6-3c2c1960c805\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " Nov 03 03:16:32.794080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.793592 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\") pod \"caf42cf6-7d10-493c-91c6-3c2c1960c805\" (UID: \"caf42cf6-7d10-493c-91c6-3c2c1960c805\") " Nov 03 03:16:32.794080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.793977 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume" pod "caf42cf6-7d10-493c-91c6-3c2c1960c805" (UID: "caf42cf6-7d10-493c-91c6-3c2c1960c805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:32.796106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.796075 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2" pod "caf42cf6-7d10-493c-91c6-3c2c1960c805" (UID: "caf42cf6-7d10-493c-91c6-3c2c1960c805"). InnerVolumeSpecName "kube-api-access-k9zx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:32.867241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.867156 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx"] Nov 03 03:16:32.870256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.870226 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56094c67-t2nsx"] Nov 03 03:16:32.894446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.894413 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zx2\" (UniqueName: \"kubernetes.io/projected/caf42cf6-7d10-493c-91c6-3c2c1960c805-kube-api-access-k9zx2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:32.894446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.894442 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf42cf6-7d10-493c-91c6-3c2c1960c805-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:32.984337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:32.984307 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017d42c1-1a69-41a5-93a7-07b4924ef2de" path="/var/lib/kubelet/pods/017d42c1-1a69-41a5-93a7-07b4924ef2de/volumes" Nov 03 03:16:33.590873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.584014 5275 generic.go:356] "Generic (PLEG): container finished" podID="caf42cf6-7d10-493c-91c6-3c2c1960c805" containerID="a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff" exitCode=0 Nov 03 03:16:33.590873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.584220 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" event={"ID":"caf42cf6-7d10-493c-91c6-3c2c1960c805","Type":"ContainerDied","Data":"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff"} Nov 03 03:16:33.590873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.584273 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" event={"ID":"caf42cf6-7d10-493c-91c6-3c2c1960c805","Type":"ContainerDied","Data":"6ed3c6881d6448b0205f780d25cae6f0b0cd88d0ce00e37330ddedbe7d0bb0f6"} Nov 03 03:16:33.590873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.584303 5275 scope.go:122] "RemoveContainer" containerID="a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff" Nov 03 03:16:33.590873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.584517 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t" Nov 03 03:16:33.607870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.604763 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" event={"ID":"29580c82-d705-4e84-a050-69f3610bec2c","Type":"ContainerStarted","Data":"cb76596e0794a64e81c485273d4e851b132dc87a308348166c3a05ca252dca83"} Nov 03 03:16:33.607870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.605461 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:33.616677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.616659 5275 scope.go:122] "RemoveContainer" containerID="a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff" Nov 03 03:16:33.617088 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:33.617067 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff\": container with ID starting with a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff not found: ID does not exist" containerID="a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff" Nov 03 03:16:33.617191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.617099 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff"} err="failed to get container status \"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff\": rpc error: code = NotFound desc = could not find container \"a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff\": container with ID starting with a70f90cb7b5dd713b6e77c3f0e1f9d81b5b94398d5b49c4d743ac73d6aa3f7ff not found: ID does not exist" Nov 03 03:16:33.752139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.752097 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:16:33.801193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.801161 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-69e611c0-7b746b47f9-hvm4t"] Nov 03 03:16:33.870480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:33.870388 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" podStartSLOduration=2.870370936 podStartE2EDuration="2.870370936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:33.868872546 +0000 UTC m=+885.021560958" watchObservedRunningTime="2026-11-03 03:16:33.870370936 +0000 UTC m=+885.023059357" Nov 03 03:16:34.983619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:34.983584 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf42cf6-7d10-493c-91c6-3c2c1960c805" path="/var/lib/kubelet/pods/caf42cf6-7d10-493c-91c6-3c2c1960c805/volumes" Nov 03 03:16:38.507606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:38.507572 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 03:16:45.620544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:45.620515 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 03:16:48.458068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:48.458013 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:16:48.458556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:48.458077 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:16:49.108307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:49.108184 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:16:49.111961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:49.111938 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:16:50.282607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.282563 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl"] Nov 03 03:16:50.328861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.328822 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e21b4958-9p9vl"] Nov 03 03:16:50.722947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.722911 5275 generic.go:356] "Generic (PLEG): container finished" podID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerID="52abbf6dfc05e17e71c48a48f49caf88ebe2c3df56285793cdf1fccb5904ed23" exitCode=0 Nov 03 03:16:50.723174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.722961 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerDied","Data":"52abbf6dfc05e17e71c48a48f49caf88ebe2c3df56285793cdf1fccb5904ed23"} Nov 03 03:16:50.723370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.723355 5275 scope.go:122] "RemoveContainer" containerID="52abbf6dfc05e17e71c48a48f49caf88ebe2c3df56285793cdf1fccb5904ed23" Nov 03 03:16:50.724566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.724533 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:50.724661 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.724621 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:50.724661 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.724654 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:50.993620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:50.993540 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d9fa89-b509-4595-8e0d-14900c01ce84" path="/var/lib/kubelet/pods/29d9fa89-b509-4595-8e0d-14900c01ce84/volumes" Nov 03 03:16:51.633666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.633627 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm"] Nov 03 03:16:51.638934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.638908 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-69e611c0-bg9lm"] Nov 03 03:16:51.708472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.708436 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:16:51.708977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.708920 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" containerID="cri-o://de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752" gracePeriod=30 Nov 03 03:16:51.737800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.737742 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerStarted","Data":"5c4cf5db2051f8886ffdadcf1b31dc45e12a98f810247177a98125a8fce6ad89"} Nov 03 03:16:51.796686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.796642 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:16:51.797204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:51.797151 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" containerID="cri-o://d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094" gracePeriod=30 Nov 03 03:16:52.196356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:52.196317 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:52.196713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:52.196691 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:52.196918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:52.196893 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:53.005757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:53.005712 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f045813-a5fa-4818-9a43-b76a3d2a9377" path="/var/lib/kubelet/pods/3f045813-a5fa-4818-9a43-b76a3d2a9377/volumes" Nov 03 03:16:53.201749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:53.201263 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:53.201749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:53.201343 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:54.196872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:54.196570 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:54.196872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:54.196647 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:55.199034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:55.198999 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:55.199602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:55.199572 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:55.504713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:55.504622 5275 patch_prober.go:29] interesting pod/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:16:55.505016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:55.504985 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:16:56.197150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:56.197064 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:56.197150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:56.197115 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:57.061350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.061226 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:16:57.089370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.089278 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:16:57.142617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142565 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.142814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142696 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.142904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142878 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.142955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142905 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142959 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.142992 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143080 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.143174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143103 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143133 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143170 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\") pod \"13791e77-2f7c-432a-8845-155113624cc3\" (UID: \"13791e77-2f7c-432a-8845-155113624cc3\") " Nov 03 03:16:57.143508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143399 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.143508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143438 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:57.144035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143606 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.144035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143629 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13791e77-2f7c-432a-8845-155113624cc3-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.144035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143646 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.144035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143659 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.144035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.143933 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.145723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.145693 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:57.145899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.145786 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "kube-api-access-28wdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:57.146060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.146040 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.146060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.146051 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo" pod "13791e77-2f7c-432a-8845-155113624cc3" (UID: "13791e77-2f7c-432a-8845-155113624cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:16:57.196804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.196765 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 03:16:57.197017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.196871 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.244801 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.244905 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.244946 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245018 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245050 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245112 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245183 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245233 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245285 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\") pod \"84db5af4-e74c-4507-aff5-675fdc33639d\" (UID: \"84db5af4-e74c-4507-aff5-675fdc33639d\") " Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245706 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wdb\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-kube-api-access-28wdb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245726 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13791e77-2f7c-432a-8845-155113624cc3-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245741 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13791e77-2f7c-432a-8845-155113624cc3-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245755 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.247163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.245770 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13791e77-2f7c-432a-8845-155113624cc3-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.248056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.247381 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:57.248224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.248194 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.248352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.248327 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.248424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.248400 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.248512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.248493 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.252209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.252167 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:57.252319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.252292 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "kube-api-access-wqrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:57.252735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.252700 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:16:57.268877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.268787 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy" pod "84db5af4-e74c-4507-aff5-675fdc33639d" (UID: "84db5af4-e74c-4507-aff5-675fdc33639d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:16:57.347167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347130 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84db5af4-e74c-4507-aff5-675fdc33639d-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347168 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347183 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347196 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347211 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrns\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-kube-api-access-wqrns\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347224 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84db5af4-e74c-4507-aff5-675fdc33639d-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347237 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84db5af4-e74c-4507-aff5-675fdc33639d-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347250 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.347513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.347262 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84db5af4-e74c-4507-aff5-675fdc33639d-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:57.821485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.821445 5275 generic.go:356] "Generic (PLEG): container finished" podID="13791e77-2f7c-432a-8845-155113624cc3" containerID="de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752" exitCode=0 Nov 03 03:16:57.821659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.821493 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" Nov 03 03:16:57.821659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.821525 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerDied","Data":"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752"} Nov 03 03:16:57.821659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.821586 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll" event={"ID":"13791e77-2f7c-432a-8845-155113624cc3","Type":"ContainerDied","Data":"606f0d550128f3265216fdb013a2d7e49656d443e60fee423f309f52b56aad0e"} Nov 03 03:16:57.821659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.821610 5275 scope.go:122] "RemoveContainer" containerID="de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752" Nov 03 03:16:57.828090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.828060 5275 generic.go:356] "Generic (PLEG): container finished" podID="84db5af4-e74c-4507-aff5-675fdc33639d" containerID="d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094" exitCode=0 Nov 03 03:16:57.828232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.828119 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" Nov 03 03:16:57.828232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.828212 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerDied","Data":"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094"} Nov 03 03:16:57.828336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.828251 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf" event={"ID":"84db5af4-e74c-4507-aff5-675fdc33639d","Type":"ContainerDied","Data":"287af8def80fbb92d2babcd1e2f061ddfcf369dd41a1d996bd8e6a74fe99188d"} Nov 03 03:16:57.834343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.834191 5275 scope.go:122] "RemoveContainer" containerID="0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c" Nov 03 03:16:57.844096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.844079 5275 scope.go:122] "RemoveContainer" containerID="de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752" Nov 03 03:16:57.844433 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:57.844407 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752\": container with ID starting with de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752 not found: ID does not exist" containerID="de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752" Nov 03 03:16:57.844500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.844447 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752"} err="failed to get container status \"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752\": rpc error: code = NotFound desc = could not find container \"de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752\": container with ID starting with de54aec47057e9b2dd8f72d752aae0dcd6e0d88e765bedc412cc54b6083f1752 not found: ID does not exist" Nov 03 03:16:57.844500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.844473 5275 scope.go:122] "RemoveContainer" containerID="0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c" Nov 03 03:16:57.844764 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:57.844745 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c\": container with ID starting with 0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c not found: ID does not exist" containerID="0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c" Nov 03 03:16:57.844805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.844772 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c"} err="failed to get container status \"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c\": rpc error: code = NotFound desc = could not find container \"0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c\": container with ID starting with 0cf58a864051eb43e941ab1d26785c7c72ce7f9634ec13373e81db47b3f4f62c not found: ID does not exist" Nov 03 03:16:57.844805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.844794 5275 scope.go:122] "RemoveContainer" containerID="d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094" Nov 03 03:16:57.854231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.854202 5275 scope.go:122] "RemoveContainer" containerID="e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed" Nov 03 03:16:57.866099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.866069 5275 scope.go:122] "RemoveContainer" containerID="d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094" Nov 03 03:16:57.866514 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:57.866485 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094\": container with ID starting with d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094 not found: ID does not exist" containerID="d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094" Nov 03 03:16:57.866607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.866526 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094"} err="failed to get container status \"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094\": rpc error: code = NotFound desc = could not find container \"d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094\": container with ID starting with d82189c13213f1fe9d3f039c97e364376cde50c7b4bb1cca31a564a2ca118094 not found: ID does not exist" Nov 03 03:16:57.866607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.866551 5275 scope.go:122] "RemoveContainer" containerID="e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed" Nov 03 03:16:57.866908 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:16:57.866883 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed\": container with ID starting with e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed not found: ID does not exist" containerID="e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed" Nov 03 03:16:57.867016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.866916 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed"} err="failed to get container status \"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed\": rpc error: code = NotFound desc = could not find container \"e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed\": container with ID starting with e5955038fa75003929a3e6e10cd36a10ba9c88cfda115dc75c2182f6a183aeed not found: ID does not exist" Nov 03 03:16:57.988932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:57.988891 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:16:58.039047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.039007 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e21b4958-openshift-default-79bb4d7577-p4gvf"] Nov 03 03:16:58.112332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.112291 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:16:58.119942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.119910 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-69e611c0-openshift-default-6bdfc4b697-t5gll"] Nov 03 03:16:58.200143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.200112 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:58.200544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.200517 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:58.201360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.201343 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:16:58.983642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.983611 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13791e77-2f7c-432a-8845-155113624cc3" path="/var/lib/kubelet/pods/13791e77-2f7c-432a-8845-155113624cc3/volumes" Nov 03 03:16:58.984087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:58.984073 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" path="/var/lib/kubelet/pods/84db5af4-e74c-4507-aff5-675fdc33639d/volumes" Nov 03 03:16:59.843707 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.843627 5275 generic.go:356] "Generic (PLEG): container finished" podID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerID="aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0" exitCode=0 Nov 03 03:16:59.844148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.843705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerDied","Data":"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0"} Nov 03 03:16:59.844205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.844190 5275 scope.go:122] "RemoveContainer" containerID="aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0" Nov 03 03:16:59.845452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.845426 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:59.845511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.845491 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:16:59.845545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:16:59.845522 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:17:00.876215 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:00.876172 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerStarted","Data":"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4"} Nov 03 03:17:01.766873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:01.762131 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:17:01.766873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:01.762300 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 03:17:01.766873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:01.762348 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 03:17:02.329641 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:17:02.329600 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:17:02.759359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:02.759262 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 03:17:02.759613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:02.759581 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 03:17:03.765728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:03.765269 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 03:17:03.765728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:03.765330 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 03:17:04.429460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:04.429432 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:17:04.769396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:04.769302 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 03:17:04.770013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:04.769978 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 03:17:05.013124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.013080 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 03:17:05.014124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014098 5275 cpu_manager.go:414] "containerMap: removing container" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.014124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014121 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014172 5275 cpu_manager.go:414] "containerMap: removing container" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014200 5275 cpu_manager.go:414] "containerMap: removing container" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014207 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014235 5275 cpu_manager.go:414] "containerMap: removing container" podUID="caf42cf6-7d10-493c-91c6-3c2c1960c805" containerName="payload-pre-processing" Nov 03 03:17:05.014295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014242 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf42cf6-7d10-493c-91c6-3c2c1960c805" containerName="payload-pre-processing" Nov 03 03:17:05.014540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014439 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.014540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014452 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf42cf6-7d10-493c-91c6-3c2c1960c805" containerName="payload-pre-processing" Nov 03 03:17:05.014540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014474 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.014540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.014492 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="13791e77-2f7c-432a-8845-155113624cc3" containerName="istio-proxy" Nov 03 03:17:05.019659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.019042 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.084719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.084671 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 03:17:05.129402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.129338 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.129647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.129625 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtx2\" (UniqueName: \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.130961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.130894 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5a6246\"" Nov 03 03:17:05.131338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.131275 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5a6246-dockercfg-z2whj\"" Nov 03 03:17:05.183903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.174407 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 03:17:05.184288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.184257 5275 cpu_manager.go:414] "containerMap: removing container" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.184391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.184292 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.184449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.184440 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="84db5af4-e74c-4507-aff5-675fdc33639d" containerName="istio-proxy" Nov 03 03:17:05.188012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.187986 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.230595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.230549 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtx2\" (UniqueName: \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.230782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.230720 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.231207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.231187 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 03:17:05.231638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.231617 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.293053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.293013 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtx2\" (UniqueName: \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\") pod \"payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.332123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.332091 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.332309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.332141 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrrw\" (UniqueName: \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.332309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.332170 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.433090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.433015 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.433407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.433347 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrrw\" (UniqueName: \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.434918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.434873 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.497997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.497961 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrrw\" (UniqueName: \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\") pod \"payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.505643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.505617 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:05.741176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.741152 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 03:17:05.743172 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:05.743144 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00de5aa_1c41_49e6_8422_0dacd79342ef.slice/crio-42fc6c6fbdecdad1924e846dfd39df9e854320271bbaa17dce0b46112f083c20 WatchSource:0}: Error finding container 42fc6c6fbdecdad1924e846dfd39df9e854320271bbaa17dce0b46112f083c20: Status 404 returned error can't find the container with id 42fc6c6fbdecdad1924e846dfd39df9e854320271bbaa17dce0b46112f083c20 Nov 03 03:17:05.757988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.757956 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 03:17:05.758151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.758015 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 03:17:05.925354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.925205 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 03:17:05.939921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.935076 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 03:17:05.939921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.935179 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:05.939921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.935211 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" event={"ID":"e00de5aa-1c41-49e6-8422-0dacd79342ef","Type":"ContainerStarted","Data":"d1909d9f29a0550e6993767d61a1697dac9e5906e6246486cf4684db72d36571"} Nov 03 03:17:05.939921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.935242 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" event={"ID":"e00de5aa-1c41-49e6-8422-0dacd79342ef","Type":"ContainerStarted","Data":"42fc6c6fbdecdad1924e846dfd39df9e854320271bbaa17dce0b46112f083c20"} Nov 03 03:17:05.939921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.935340 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:05.956756 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:05.956722 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa85c3b_27e9_4417_8c72_32cf5a365ad5.slice/crio-bc62d3b1651658a133a30418c30666f05c4762cd29c1a70a37f1410ecd008741 WatchSource:0}: Error finding container bc62d3b1651658a133a30418c30666f05c4762cd29c1a70a37f1410ecd008741: Status 404 returned error can't find the container with id bc62d3b1651658a133a30418c30666f05c4762cd29c1a70a37f1410ecd008741 Nov 03 03:17:05.980623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.980573 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 03:17:05.993068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.992188 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"simulator-w2-de27f5-kserve-self-signed-certs\"" Nov 03 03:17:05.993068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.992258 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"kube-root-ca.crt\"" Nov 03 03:17:05.993068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.992325 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"openshift-service-ca.crt\"" Nov 03 03:17:05.993689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:05.993666 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"default-dockercfg-fqsgm\"" Nov 03 03:17:06.050543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.046206 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.050543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.046438 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.050543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.046811 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.050543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.046925 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.050543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.046958 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.053801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.053770 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwfd\" (UniqueName: \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.067252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.067184 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" podStartSLOduration=2.067158309 podStartE2EDuration="2.067158309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:06.054519854 +0000 UTC m=+917.207208278" watchObservedRunningTime="2026-11-03 03:17:06.067158309 +0000 UTC m=+917.219846731" Nov 03 03:17:06.155593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155496 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.155593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155539 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.155593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155575 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.155889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155701 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.155889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155753 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwfd\" (UniqueName: \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.155889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.155862 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.156277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.156165 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.156277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.156201 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.156438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.156333 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.157942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.157917 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.158510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.158492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.217384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.217345 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwfd\" (UniqueName: \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\") pod \"simulator-w2-de27f5-kserve-56bd87cc7d-gptrn\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.257086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.257046 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:06.664152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.664120 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 03:17:06.665641 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:06.665607 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod803d934a_5742_443b_886a_7420addbd746.slice/crio-40f05007e2bec4f63716c061f171c1b62978d7aa3914469cbd119967398935a9 WatchSource:0}: Error finding container 40f05007e2bec4f63716c061f171c1b62978d7aa3914469cbd119967398935a9: Status 404 returned error can't find the container with id 40f05007e2bec4f63716c061f171c1b62978d7aa3914469cbd119967398935a9 Nov 03 03:17:06.761746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.761717 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:17:06.762647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.762628 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:17:06.939561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.939476 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" event={"ID":"eaa85c3b-27e9-4417-8c72-32cf5a365ad5","Type":"ContainerStarted","Data":"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5"} Nov 03 03:17:06.939561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.939540 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:06.940063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.939568 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" event={"ID":"eaa85c3b-27e9-4417-8c72-32cf5a365ad5","Type":"ContainerStarted","Data":"bc62d3b1651658a133a30418c30666f05c4762cd29c1a70a37f1410ecd008741"} Nov 03 03:17:06.941264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.941240 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" event={"ID":"803d934a-5742-443b-886a-7420addbd746","Type":"ContainerStarted","Data":"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3"} Nov 03 03:17:06.941383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:06.941275 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" event={"ID":"803d934a-5742-443b-886a-7420addbd746","Type":"ContainerStarted","Data":"40f05007e2bec4f63716c061f171c1b62978d7aa3914469cbd119967398935a9"} Nov 03 03:17:07.266398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.261157 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:07.266725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.266663 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" podStartSLOduration=2.266640421 podStartE2EDuration="2.266640421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:07.133134288 +0000 UTC m=+918.285822711" watchObservedRunningTime="2026-11-03 03:17:07.266640421 +0000 UTC m=+918.419328839" Nov 03 03:17:07.267811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.267752 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podStartSLOduration=2.267736223 podStartE2EDuration="2.267736223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:07.258126444 +0000 UTC m=+918.410814888" watchObservedRunningTime="2026-11-03 03:17:07.267736223 +0000 UTC m=+918.420424645" Nov 03 03:17:07.553384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.553345 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 03:17:07.558933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.558897 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.644461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.629391 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b1ad4e-dockercfg-ljflw\"" Nov 03 03:17:07.644461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.629509 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b1ad4e\"" Nov 03 03:17:07.644461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.633113 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 03:17:07.676765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.676731 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.676972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.676810 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.685774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.685744 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 03:17:07.689517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.689499 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.747673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.747639 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 03:17:07.778100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.778040 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.778305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.778152 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.778305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.778183 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.778305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.778265 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w672n\" (UniqueName: \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.779140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.779107 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.839548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.839481 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\") pod \"payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.873361 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.873322 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:07.879163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.879141 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w672n\" (UniqueName: \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.879284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.879255 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.879950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.879931 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:07.940944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:07.940916 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w672n\" (UniqueName: \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\") pod \"payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:08.004002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.003966 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:08.249176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.249152 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 03:17:08.251386 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:08.251356 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952569b9_4a82_4650_82e7_79de667bbd6f.slice/crio-aa54e8fdba86e5a993e76bc33a7e92d2fc118d16a7f444424ff70bab6389a900 WatchSource:0}: Error finding container aa54e8fdba86e5a993e76bc33a7e92d2fc118d16a7f444424ff70bab6389a900: Status 404 returned error can't find the container with id aa54e8fdba86e5a993e76bc33a7e92d2fc118d16a7f444424ff70bab6389a900 Nov 03 03:17:08.382043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.382019 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 03:17:08.384038 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:08.384005 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef707fe_4249_4ad5_a27f_265370cb9ef3.slice/crio-f48b4a4f0d7bbb17990b979410f9a98f9a348bd5be0d4598aef908ea9601a00b WatchSource:0}: Error finding container f48b4a4f0d7bbb17990b979410f9a98f9a348bd5be0d4598aef908ea9601a00b: Status 404 returned error can't find the container with id f48b4a4f0d7bbb17990b979410f9a98f9a348bd5be0d4598aef908ea9601a00b Nov 03 03:17:08.954150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.954113 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" event={"ID":"5ef707fe-4249-4ad5-a27f-265370cb9ef3","Type":"ContainerStarted","Data":"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65"} Nov 03 03:17:08.954603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.954176 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" event={"ID":"5ef707fe-4249-4ad5-a27f-265370cb9ef3","Type":"ContainerStarted","Data":"f48b4a4f0d7bbb17990b979410f9a98f9a348bd5be0d4598aef908ea9601a00b"} Nov 03 03:17:08.954603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.954301 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:08.955868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.955826 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" event={"ID":"952569b9-4a82-4650-82e7-79de667bbd6f","Type":"ContainerStarted","Data":"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961"} Nov 03 03:17:08.956000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.955884 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" event={"ID":"952569b9-4a82-4650-82e7-79de667bbd6f","Type":"ContainerStarted","Data":"aa54e8fdba86e5a993e76bc33a7e92d2fc118d16a7f444424ff70bab6389a900"} Nov 03 03:17:08.956000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:08.955918 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:09.073679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:09.073626 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" podStartSLOduration=2.073610734 podStartE2EDuration="2.073610734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:09.070834002 +0000 UTC m=+920.223522433" watchObservedRunningTime="2026-11-03 03:17:09.073610734 +0000 UTC m=+920.226299196" Nov 03 03:17:09.188882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:09.188792 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" podStartSLOduration=2.188772785 podStartE2EDuration="2.188772785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:09.186543006 +0000 UTC m=+920.339231428" watchObservedRunningTime="2026-11-03 03:17:09.188772785 +0000 UTC m=+920.341461208" Nov 03 03:17:16.257247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:16.257206 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:16.269913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:16.269888 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:16.944808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:16.944781 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 03:17:17.017298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:17.017272 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:17:17.950078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:17.950052 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 03:17:19.964693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:19.964663 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 03:17:19.965151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:19.964740 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 03:17:31.666378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.666335 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 03:17:31.670761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.670738 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.728810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.728762 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.729039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.728875 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.729039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.728956 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.729039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.728998 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.729229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.729037 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cwh\" (UniqueName: \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.729229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.729118 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.774554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.774518 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"simulator-w3-68b670-kserve-self-signed-certs\"" Nov 03 03:17:31.774554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.774544 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"default-dockercfg-x2m5k\"" Nov 03 03:17:31.774554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.774551 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"openshift-service-ca.crt\"" Nov 03 03:17:31.774879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.774618 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"kube-root-ca.crt\"" Nov 03 03:17:31.776469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.776448 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 03:17:31.830026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.829994 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830032 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830060 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cwh\" (UniqueName: \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830104 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830135 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830168 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830463 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830534 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.830683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.830637 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.832581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.832556 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.832811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.832797 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.891970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.891942 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cwh\" (UniqueName: \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\") pod \"simulator-w3-68b670-kserve-6bd75776f7-nvww9\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:31.985885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:31.985759 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:32.330662 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:17:32.330628 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:17:32.371899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:32.371872 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 03:17:32.373741 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:32.373713 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027de7ab_0a79_45fd_afce_1dee74384a53.slice/crio-c40a6dc1a6c65150b47e14ef1fb93a96f846cbfaa136531f0f7120e4fce35903 WatchSource:0}: Error finding container c40a6dc1a6c65150b47e14ef1fb93a96f846cbfaa136531f0f7120e4fce35903: Status 404 returned error can't find the container with id c40a6dc1a6c65150b47e14ef1fb93a96f846cbfaa136531f0f7120e4fce35903 Nov 03 03:17:33.101594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:33.101558 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" event={"ID":"027de7ab-0a79-45fd-afce-1dee74384a53","Type":"ContainerStarted","Data":"d91a840355f9ed9107fddacc8f59357348acca13c885153437ac45f23186e69b"} Nov 03 03:17:33.102046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:33.101603 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" event={"ID":"027de7ab-0a79-45fd-afce-1dee74384a53","Type":"ContainerStarted","Data":"c40a6dc1a6c65150b47e14ef1fb93a96f846cbfaa136531f0f7120e4fce35903"} Nov 03 03:17:33.222561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:33.222508 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podStartSLOduration=2.222494178 podStartE2EDuration="2.222494178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:33.221064164 +0000 UTC m=+944.373752586" watchObservedRunningTime="2026-11-03 03:17:33.222494178 +0000 UTC m=+944.375182599" Nov 03 03:17:33.986556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:33.986522 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:41.986815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:41.986776 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:41.999509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:41.999483 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:42.173699 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:42.173672 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:17:44.967636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:44.967597 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 03:17:44.972784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:44.972759 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.077122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.077082 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"premium-w2-de27f5-kserve-self-signed-certs\"" Nov 03 03:17:45.079299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.079279 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 03:17:45.086548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086525 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.086593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086569 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdj4p\" (UniqueName: \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.086777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086759 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.086818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086792 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.086950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086935 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.087009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.086999 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188275 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188318 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdj4p\" (UniqueName: \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188353 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188378 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188443 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188524 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188837 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188915 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.188983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.188915 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.191018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.190997 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.191132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.191113 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.252269 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.252200 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdj4p\" (UniqueName: \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\") pod \"premium-w2-de27f5-kserve-6dc56cbf47-s7hrw\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.286223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.286186 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:45.668384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:45.668359 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 03:17:45.669839 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:17:45.669816 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b18951d_2a00_4f58_a3d0_5451c61ec179.slice/crio-26cb31bff94e1042c9090f1078fe2011d45f592c60bcaad6092bc498a1d1fccb WatchSource:0}: Error finding container 26cb31bff94e1042c9090f1078fe2011d45f592c60bcaad6092bc498a1d1fccb: Status 404 returned error can't find the container with id 26cb31bff94e1042c9090f1078fe2011d45f592c60bcaad6092bc498a1d1fccb Nov 03 03:17:46.193960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:46.193915 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" event={"ID":"7b18951d-2a00-4f58-a3d0-5451c61ec179","Type":"ContainerStarted","Data":"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa"} Nov 03 03:17:46.194400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:46.193974 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" event={"ID":"7b18951d-2a00-4f58-a3d0-5451c61ec179","Type":"ContainerStarted","Data":"26cb31bff94e1042c9090f1078fe2011d45f592c60bcaad6092bc498a1d1fccb"} Nov 03 03:17:46.286718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:46.286687 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:46.315071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:46.315013 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podStartSLOduration=2.314999222 podStartE2EDuration="2.314999222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:46.312383416 +0000 UTC m=+957.465071834" watchObservedRunningTime="2026-11-03 03:17:46.314999222 +0000 UTC m=+957.467687642" Nov 03 03:17:55.287337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:55.287299 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:55.300239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:55.300207 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:17:56.264965 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:17:56.264937 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:18:02.330642 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:18:02.330597 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:18:32.332751 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:18:32.332658 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:18:35.378225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.378190 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 03:18:35.382087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.382064 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.434540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.434508 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"kube-root-ca.crt\"" Nov 03 03:18:35.443931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.443901 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwnl\" (UniqueName: \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.444117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.443963 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.444185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.444087 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.444240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.444216 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.444287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.444258 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.444704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.444674 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.483899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.483833 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"openshift-service-ca.crt\"" Nov 03 03:18:35.484136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.483990 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"simulator-w4-b1ad4e-kserve-self-signed-certs\"" Nov 03 03:18:35.484203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.484046 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"default-dockercfg-qtmq4\"" Nov 03 03:18:35.487063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.487036 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 03:18:35.546373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546336 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.546680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546397 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwnl\" (UniqueName: \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.546680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546460 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.546680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546493 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.546680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546541 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.546680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.546557 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.547036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.547014 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.547243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.547226 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.548159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.548075 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.550075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.550049 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.550225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.550208 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.608605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.608570 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwnl\" (UniqueName: \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\") pod \"simulator-w4-b1ad4e-kserve-d66677d94-wj5xc\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:35.694556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:35.694449 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:36.073103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:36.073074 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 03:18:36.075168 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:18:36.075136 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20f01655_4ce7_4fbe_aa5d_8cbcbfc9a3af.slice/crio-960214dd651f2edba2c8171506f6c92a1a2868b8fe5fd84697cf8956d24f4373 WatchSource:0}: Error finding container 960214dd651f2edba2c8171506f6c92a1a2868b8fe5fd84697cf8956d24f4373: Status 404 returned error can't find the container with id 960214dd651f2edba2c8171506f6c92a1a2868b8fe5fd84697cf8956d24f4373 Nov 03 03:18:36.509264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:36.509175 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" event={"ID":"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af","Type":"ContainerStarted","Data":"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a"} Nov 03 03:18:36.509264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:36.509217 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" event={"ID":"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af","Type":"ContainerStarted","Data":"960214dd651f2edba2c8171506f6c92a1a2868b8fe5fd84697cf8956d24f4373"} Nov 03 03:18:36.627685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:36.627619 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podStartSLOduration=1.62759798 podStartE2EDuration="1.62759798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:36.627174654 +0000 UTC m=+1007.779863075" watchObservedRunningTime="2026-11-03 03:18:36.62759798 +0000 UTC m=+1007.780286415" Nov 03 03:18:36.695348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:36.695317 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:44.590647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.590606 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 03:18:44.594794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.594773 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646095 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646234 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646323 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmvt\" (UniqueName: \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646360 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.646491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.646478 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.653202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.653174 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 03:18:44.699645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.699611 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"premium-w3-68b670-kserve-self-signed-certs\"" Nov 03 03:18:44.747090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747056 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmvt\" (UniqueName: \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747119 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747159 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747193 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747327 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747458 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747663 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747729 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.747825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.747751 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.749637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.749615 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.750238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.750223 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.808543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.808508 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmvt\" (UniqueName: \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\") pod \"premium-w3-68b670-kserve-7fb796b996-qvl9p\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:44.907698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:44.907610 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:45.298202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.298178 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 03:18:45.300463 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:18:45.300434 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99758ef7_4c84_451c_ba20_8b437d442785.slice/crio-a037ff314ba475c0615dcff94bafe049501d6ba768cd21139805ce07273b98ed WatchSource:0}: Error finding container a037ff314ba475c0615dcff94bafe049501d6ba768cd21139805ce07273b98ed: Status 404 returned error can't find the container with id a037ff314ba475c0615dcff94bafe049501d6ba768cd21139805ce07273b98ed Nov 03 03:18:45.567430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.567298 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" event={"ID":"99758ef7-4c84-451c-ba20-8b437d442785","Type":"ContainerStarted","Data":"19371ecc43380e4df31eafb37211f1839c926eb6cc3e13ef7cb4a910ce5f3b9e"} Nov 03 03:18:45.567430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.567349 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" event={"ID":"99758ef7-4c84-451c-ba20-8b437d442785","Type":"ContainerStarted","Data":"a037ff314ba475c0615dcff94bafe049501d6ba768cd21139805ce07273b98ed"} Nov 03 03:18:45.684425 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.684353 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podStartSLOduration=1.684335585 podStartE2EDuration="1.684335585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:45.683402273 +0000 UTC m=+1016.836090693" watchObservedRunningTime="2026-11-03 03:18:45.684335585 +0000 UTC m=+1016.837024008" Nov 03 03:18:45.694781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.694756 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:45.707517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.707494 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:45.907985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:45.907899 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:46.585879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:46.585810 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:18:54.908341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:54.908307 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:54.921905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:54.921870 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:55.641707 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:55.641671 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:18:56.986272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:56.986242 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc"] Nov 03 03:18:56.990178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:56.990157 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.049215 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.049156 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc"] Nov 03 03:18:57.086382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.086346 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.086541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.086390 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.086612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.086582 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj6q\" (UniqueName: \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.094960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.094925 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:18:57.095112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.095095 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:18:57.095189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.095136 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:18:57.095189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.095134 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:18:57.188250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.188209 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.188250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.188257 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.188521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.188323 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj6q\" (UniqueName: \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.188971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.188950 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.190705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.190679 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.250758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.250682 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj6q\" (UniqueName: \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\") pod \"maas-api-revoke-sub-59ad2098-wfwlc\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.303350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.303319 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:57.687000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:57.686973 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc"] Nov 03 03:18:57.688863 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:18:57.688815 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e98d58_f759_4dd3_89d2_4b79b75d7914.slice/crio-26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f WatchSource:0}: Error finding container 26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f: Status 404 returned error can't find the container with id 26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f Nov 03 03:18:58.651417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:58.651376 5275 generic.go:356] "Generic (PLEG): container finished" podID="31e98d58-f759-4dd3-89d2-4b79b75d7914" containerID="fe58c7a5db8bf673023ce411959c8ab0d81ed7f4bd22191d17ea9a2f526ab9fb" exitCode=0 Nov 03 03:18:58.651826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:58.651414 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" event={"ID":"31e98d58-f759-4dd3-89d2-4b79b75d7914","Type":"ContainerDied","Data":"fe58c7a5db8bf673023ce411959c8ab0d81ed7f4bd22191d17ea9a2f526ab9fb"} Nov 03 03:18:58.651826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:58.651471 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" event={"ID":"31e98d58-f759-4dd3-89d2-4b79b75d7914","Type":"ContainerStarted","Data":"26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f"} Nov 03 03:18:59.822549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.822519 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:18:59.916489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.916400 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\") pod \"31e98d58-f759-4dd3-89d2-4b79b75d7914\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " Nov 03 03:18:59.916489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.916440 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\") pod \"31e98d58-f759-4dd3-89d2-4b79b75d7914\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " Nov 03 03:18:59.916725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.916538 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\" (UniqueName: \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\") pod \"31e98d58-f759-4dd3-89d2-4b79b75d7914\" (UID: \"31e98d58-f759-4dd3-89d2-4b79b75d7914\") " Nov 03 03:18:59.916918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.916890 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca" pod "31e98d58-f759-4dd3-89d2-4b79b75d7914" (UID: "31e98d58-f759-4dd3-89d2-4b79b75d7914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:18:59.918623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.918603 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca" pod "31e98d58-f759-4dd3-89d2-4b79b75d7914" (UID: "31e98d58-f759-4dd3-89d2-4b79b75d7914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:18:59.918842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:18:59.918815 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q" pod "31e98d58-f759-4dd3-89d2-4b79b75d7914" (UID: "31e98d58-f759-4dd3-89d2-4b79b75d7914"). InnerVolumeSpecName "kube-api-access-mzj6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:19:00.017615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.017578 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:00.017615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.017609 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e98d58-f759-4dd3-89d2-4b79b75d7914-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:00.017615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.017620 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzj6q\" (UniqueName: \"kubernetes.io/projected/31e98d58-f759-4dd3-89d2-4b79b75d7914-kube-api-access-mzj6q\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:00.665011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.664970 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" event={"ID":"31e98d58-f759-4dd3-89d2-4b79b75d7914","Type":"ContainerDied","Data":"26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f"} Nov 03 03:19:00.665222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.665032 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f9537ac900da9307357e3d9612a088db89df1f4fb4c17873a91594af5d424f" Nov 03 03:19:00.665222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:00.664985 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc" Nov 03 03:19:02.330236 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:19:02.330204 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:19:15.081581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.081544 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97"] Nov 03 03:19:15.082423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.082027 5275 cpu_manager.go:414] "containerMap: removing container" podUID="31e98d58-f759-4dd3-89d2-4b79b75d7914" containerName="revoke-keys" Nov 03 03:19:15.082423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.082039 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e98d58-f759-4dd3-89d2-4b79b75d7914" containerName="revoke-keys" Nov 03 03:19:15.082423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.082126 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e98d58-f759-4dd3-89d2-4b79b75d7914" containerName="revoke-keys" Nov 03 03:19:15.085414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.085394 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.178095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.178044 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.178486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.178460 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gjv\" (UniqueName: \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.178711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.178684 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.190253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.190215 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:19:15.190438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.190284 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:19:15.190438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.190291 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:19:15.190438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.190373 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:19:15.194211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.194183 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97"] Nov 03 03:19:15.280291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.280259 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.280497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.280366 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.280497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.280422 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gjv\" (UniqueName: \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.281179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.281152 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.283007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.282983 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.341750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.341676 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gjv\" (UniqueName: \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\") pod \"maas-api-revoke-sub-21587eec-w7z97\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.398459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.398423 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:15.779670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:15.779642 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97"] Nov 03 03:19:15.781704 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:19:15.781673 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97010889_70a4_4aef_8810_8ee36c1bda6d.slice/crio-a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5 WatchSource:0}: Error finding container a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5: Status 404 returned error can't find the container with id a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5 Nov 03 03:19:16.767911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:16.767869 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" event={"ID":"97010889-70a4-4aef-8810-8ee36c1bda6d","Type":"ContainerStarted","Data":"d6f8496c790ea22ad097ed25983d9542882ea3b889ec28eab8203e35f5028100"} Nov 03 03:19:16.768374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:16.767937 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" event={"ID":"97010889-70a4-4aef-8810-8ee36c1bda6d","Type":"ContainerStarted","Data":"a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5"} Nov 03 03:19:16.887463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:16.887392 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" podStartSLOduration=1.887371233 podStartE2EDuration="1.887371233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:16.883134134 +0000 UTC m=+1048.035822556" watchObservedRunningTime="2026-11-03 03:19:16.887371233 +0000 UTC m=+1048.040059656" Nov 03 03:19:17.777755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:17.777716 5275 generic.go:356] "Generic (PLEG): container finished" podID="97010889-70a4-4aef-8810-8ee36c1bda6d" containerID="d6f8496c790ea22ad097ed25983d9542882ea3b889ec28eab8203e35f5028100" exitCode=0 Nov 03 03:19:17.778271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:17.777791 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" event={"ID":"97010889-70a4-4aef-8810-8ee36c1bda6d","Type":"ContainerDied","Data":"d6f8496c790ea22ad097ed25983d9542882ea3b889ec28eab8203e35f5028100"} Nov 03 03:19:18.929888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:18.929839 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:19.021964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.021935 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\" (UniqueName: \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\") pod \"97010889-70a4-4aef-8810-8ee36c1bda6d\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " Nov 03 03:19:19.021964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.021974 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\") pod \"97010889-70a4-4aef-8810-8ee36c1bda6d\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " Nov 03 03:19:19.022211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.022009 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\") pod \"97010889-70a4-4aef-8810-8ee36c1bda6d\" (UID: \"97010889-70a4-4aef-8810-8ee36c1bda6d\") " Nov 03 03:19:19.022412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.022382 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca" pod "97010889-70a4-4aef-8810-8ee36c1bda6d" (UID: "97010889-70a4-4aef-8810-8ee36c1bda6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:19:19.024311 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.024286 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca" pod "97010889-70a4-4aef-8810-8ee36c1bda6d" (UID: "97010889-70a4-4aef-8810-8ee36c1bda6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:19:19.024427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.024308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv" pod "97010889-70a4-4aef-8810-8ee36c1bda6d" (UID: "97010889-70a4-4aef-8810-8ee36c1bda6d"). InnerVolumeSpecName "kube-api-access-n4gjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:19:19.122951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.122837 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gjv\" (UniqueName: \"kubernetes.io/projected/97010889-70a4-4aef-8810-8ee36c1bda6d-kube-api-access-n4gjv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:19.122951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.122894 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:19.122951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.122904 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97010889-70a4-4aef-8810-8ee36c1bda6d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:19.791244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.791213 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" Nov 03 03:19:19.791244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.791221 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97" event={"ID":"97010889-70a4-4aef-8810-8ee36c1bda6d","Type":"ContainerDied","Data":"a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5"} Nov 03 03:19:19.791473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:19.791274 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99445006025cc1fa710239c02f5cd806aa160a9c1a3a7bc3b063be6423308d5" Nov 03 03:19:32.332891 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:19:32.332836 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:19:41.603125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.603087 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 03:19:41.603786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.603768 5275 cpu_manager.go:414] "containerMap: removing container" podUID="97010889-70a4-4aef-8810-8ee36c1bda6d" containerName="revoke-keys" Nov 03 03:19:41.603840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.603790 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97010889-70a4-4aef-8810-8ee36c1bda6d" containerName="revoke-keys" Nov 03 03:19:41.603987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.603974 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="97010889-70a4-4aef-8810-8ee36c1bda6d" containerName="revoke-keys" Nov 03 03:19:41.607772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.607754 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689468 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689576 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689637 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pl6\" (UniqueName: \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689736 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.689907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.689786 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.710549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.710519 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"premium-w4-b1ad4e-kserve-self-signed-certs\"" Nov 03 03:19:41.713275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.713250 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 03:19:41.790621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790588 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.790831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790631 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.790831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790672 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.790831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790688 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pl6\" (UniqueName: \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.790831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790709 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.790831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.790749 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.791105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.791074 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.791251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.791227 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.791251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.791242 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.793337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.793312 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.793617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.793595 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.852719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.852660 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pl6\" (UniqueName: \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\") pod \"premium-w4-b1ad4e-kserve-d8d94b579-mvzn4\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:41.920894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:41.920770 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:42.341304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:42.341279 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 03:19:42.343413 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:19:42.343375 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod456f1b37_ed97_4514_932c_141a33ade80e.slice/crio-d649b3f5d7e5c8475da803285c0704d798b80e75044b13b746a5782e4c8de028 WatchSource:0}: Error finding container d649b3f5d7e5c8475da803285c0704d798b80e75044b13b746a5782e4c8de028: Status 404 returned error can't find the container with id d649b3f5d7e5c8475da803285c0704d798b80e75044b13b746a5782e4c8de028 Nov 03 03:19:42.958507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:42.958471 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" event={"ID":"456f1b37-ed97-4514-932c-141a33ade80e","Type":"ContainerStarted","Data":"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0"} Nov 03 03:19:42.959039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:42.958518 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" event={"ID":"456f1b37-ed97-4514-932c-141a33ade80e","Type":"ContainerStarted","Data":"d649b3f5d7e5c8475da803285c0704d798b80e75044b13b746a5782e4c8de028"} Nov 03 03:19:43.080008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:43.079957 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podStartSLOduration=2.079943065 podStartE2EDuration="2.079943065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:43.078632916 +0000 UTC m=+1074.231321338" watchObservedRunningTime="2026-11-03 03:19:43.079943065 +0000 UTC m=+1074.232631487" Nov 03 03:19:43.921767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:43.921732 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:46.224814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.224774 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8"] Nov 03 03:19:46.229056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.229033 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.241122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.241099 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lzb\" (UniqueName: \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.241244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.241226 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.241391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.241378 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.283239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.283204 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:19:46.332795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.332762 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:19:46.335608 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.335588 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8"] Nov 03 03:19:46.342783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.342762 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.342903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.342886 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.342952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.342917 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lzb\" (UniqueName: \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.345379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.345349 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.364509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.364477 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:19:46.364667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.364554 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:19:46.374254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.374231 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.405158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.405126 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lzb\" (UniqueName: \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\") pod \"maas-api-revoke-sub-ef89d962-vfsq8\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.543306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.543272 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:46.923232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.923207 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8"] Nov 03 03:19:46.925841 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:19:46.925793 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82abd35a_0fc4_451f_9db7_a96923917f82.slice/crio-a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c WatchSource:0}: Error finding container a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c: Status 404 returned error can't find the container with id a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c Nov 03 03:19:46.992021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:46.987050 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" event={"ID":"82abd35a-0fc4-451f-9db7-a96923917f82","Type":"ContainerStarted","Data":"a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c"} Nov 03 03:19:47.994700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:47.994660 5275 generic.go:356] "Generic (PLEG): container finished" podID="82abd35a-0fc4-451f-9db7-a96923917f82" containerID="390c7149679e046245dfb4ce83c76eb8eb4e613d84da84398e5e8612e8cd8f40" exitCode=0 Nov 03 03:19:47.995174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:47.994748 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" event={"ID":"82abd35a-0fc4-451f-9db7-a96923917f82","Type":"ContainerDied","Data":"390c7149679e046245dfb4ce83c76eb8eb4e613d84da84398e5e8612e8cd8f40"} Nov 03 03:19:49.155212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.155182 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:49.174212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.174173 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\") pod \"82abd35a-0fc4-451f-9db7-a96923917f82\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " Nov 03 03:19:49.174402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.174243 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\" (UniqueName: \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\") pod \"82abd35a-0fc4-451f-9db7-a96923917f82\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " Nov 03 03:19:49.174402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.174283 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\") pod \"82abd35a-0fc4-451f-9db7-a96923917f82\" (UID: \"82abd35a-0fc4-451f-9db7-a96923917f82\") " Nov 03 03:19:49.174909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.174841 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca" pod "82abd35a-0fc4-451f-9db7-a96923917f82" (UID: "82abd35a-0fc4-451f-9db7-a96923917f82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:19:49.176690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.176661 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb" pod "82abd35a-0fc4-451f-9db7-a96923917f82" (UID: "82abd35a-0fc4-451f-9db7-a96923917f82"). InnerVolumeSpecName "kube-api-access-r7lzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:19:49.176931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.176899 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca" pod "82abd35a-0fc4-451f-9db7-a96923917f82" (UID: "82abd35a-0fc4-451f-9db7-a96923917f82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:19:49.275838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.275790 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:49.275838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.275837 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lzb\" (UniqueName: \"kubernetes.io/projected/82abd35a-0fc4-451f-9db7-a96923917f82-kube-api-access-r7lzb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:49.276103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:49.275873 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82abd35a-0fc4-451f-9db7-a96923917f82-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:19:50.008192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:50.008159 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" Nov 03 03:19:50.008192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:50.008179 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8" event={"ID":"82abd35a-0fc4-451f-9db7-a96923917f82","Type":"ContainerDied","Data":"a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c"} Nov 03 03:19:50.008420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:50.008222 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a712c8c22378f9ee9d634fd5ba5912b6050ad961069b67ae84de0442f75d836c" Nov 03 03:19:51.921611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:51.921546 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:51.934978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:51.934948 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:19:52.084873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:19:52.084138 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:20:00.019258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.019216 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h"] Nov 03 03:20:00.019743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.019729 5275 cpu_manager.go:414] "containerMap: removing container" podUID="82abd35a-0fc4-451f-9db7-a96923917f82" containerName="revoke-keys" Nov 03 03:20:00.019784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.019744 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82abd35a-0fc4-451f-9db7-a96923917f82" containerName="revoke-keys" Nov 03 03:20:00.019841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.019831 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="82abd35a-0fc4-451f-9db7-a96923917f82" containerName="revoke-keys" Nov 03 03:20:00.023158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.023140 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.081993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.081956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h"] Nov 03 03:20:00.105828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.105793 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6kv\" (UniqueName: \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.106037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.105986 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.106205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.106180 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.127004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.126972 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:20:00.127186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.127049 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:20:00.127186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.127104 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:20:00.127186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.127174 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:20:00.207161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.207126 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.207343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.207227 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.207343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.207285 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6kv\" (UniqueName: \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.207968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.207949 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.209685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.209668 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.268935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.268903 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6kv\" (UniqueName: \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\") pod \"maas-api-revoke-sub-8e3fc7d7-c994h\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.336415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.336377 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:00.715634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:00.715607 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h"] Nov 03 03:20:00.717988 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:00.717952 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf782a5ae_1f3c_4c4f_a10b_e51c5fe5d9c3.slice/crio-4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4 WatchSource:0}: Error finding container 4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4: Status 404 returned error can't find the container with id 4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4 Nov 03 03:20:01.092716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:01.092684 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" event={"ID":"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3","Type":"ContainerStarted","Data":"8c4c94a784417367dfe3ac143772315be1463efd43e1e2b283d1cd445dbe2c04"} Nov 03 03:20:01.093189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:01.092735 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" event={"ID":"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3","Type":"ContainerStarted","Data":"4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4"} Nov 03 03:20:01.212895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:01.212820 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" podStartSLOduration=2.212803482 podStartE2EDuration="2.212803482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:01.209945831 +0000 UTC m=+1092.362634253" watchObservedRunningTime="2026-11-03 03:20:01.212803482 +0000 UTC m=+1092.365491902" Nov 03 03:20:01.991900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:01.991793 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s"] Nov 03 03:20:01.995682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:01.995659 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.053693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.053645 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s"] Nov 03 03:20:02.108450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.108408 5275 generic.go:356] "Generic (PLEG): container finished" podID="f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" containerID="8c4c94a784417367dfe3ac143772315be1463efd43e1e2b283d1cd445dbe2c04" exitCode=0 Nov 03 03:20:02.108975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.108497 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" event={"ID":"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3","Type":"ContainerDied","Data":"8c4c94a784417367dfe3ac143772315be1463efd43e1e2b283d1cd445dbe2c04"} Nov 03 03:20:02.128400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.128362 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pqc\" (UniqueName: \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.128400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.128408 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.128657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.128622 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.229898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.229859 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pqc\" (UniqueName: \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.230053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.229919 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.230131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.230114 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.230765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.230743 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.232536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.232505 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.291717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.291681 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pqc\" (UniqueName: \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\") pod \"maas-api-revoke-sub-877156c9-lk55s\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.309573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.309545 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:02.335451 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:20:02.335414 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:20:02.690902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:02.690838 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s"] Nov 03 03:20:02.692646 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:02.692616 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f7d56e_56b7_4b04_9cc5_de7d8a5325b1.slice/crio-55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5 WatchSource:0}: Error finding container 55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5: Status 404 returned error can't find the container with id 55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5 Nov 03 03:20:03.116716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.116678 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" event={"ID":"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1","Type":"ContainerStarted","Data":"fabddb54d09031992de79df675e932bc0abc8d10926d5aa1c638db2c2d55b62c"} Nov 03 03:20:03.117208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.116743 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" event={"ID":"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1","Type":"ContainerStarted","Data":"55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5"} Nov 03 03:20:03.234422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.234375 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" podStartSLOduration=2.234360305 podStartE2EDuration="2.234360305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:03.231957501 +0000 UTC m=+1094.384645921" watchObservedRunningTime="2026-11-03 03:20:03.234360305 +0000 UTC m=+1094.387048725" Nov 03 03:20:03.261371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.261347 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:03.341929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.341895 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\") pod \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " Nov 03 03:20:03.342138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.342013 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\") pod \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " Nov 03 03:20:03.342138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.342118 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\" (UniqueName: \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\") pod \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\" (UID: \"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3\") " Nov 03 03:20:03.342361 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.342333 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca" pod "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" (UID: "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:03.344429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.344396 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca" pod "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" (UID: "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:03.344532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.344472 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv" pod "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" (UID: "f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3"). InnerVolumeSpecName "kube-api-access-fd6kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:03.443959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.443927 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6kv\" (UniqueName: \"kubernetes.io/projected/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-kube-api-access-fd6kv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:03.443959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.443954 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:03.443959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:03.443965 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:04.124415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:04.124371 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" Nov 03 03:20:04.124415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:04.124402 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h" event={"ID":"f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3","Type":"ContainerDied","Data":"4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4"} Nov 03 03:20:04.124937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:04.124455 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c51de46735ca75f04ffbc615f0f473b751c05fc7784159068d661cc99eb86f4" Nov 03 03:20:04.126153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:04.126130 5275 generic.go:356] "Generic (PLEG): container finished" podID="e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" containerID="fabddb54d09031992de79df675e932bc0abc8d10926d5aa1c638db2c2d55b62c" exitCode=0 Nov 03 03:20:04.126266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:04.126161 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" event={"ID":"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1","Type":"ContainerDied","Data":"fabddb54d09031992de79df675e932bc0abc8d10926d5aa1c638db2c2d55b62c"} Nov 03 03:20:05.283186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.283153 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:05.366694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.366655 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\") pod \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " Nov 03 03:20:05.366925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.366787 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\") pod \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " Nov 03 03:20:05.366925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.366828 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\" (UniqueName: \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\") pod \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\" (UID: \"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1\") " Nov 03 03:20:05.367177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.367148 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca" pod "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" (UID: "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:05.367302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.367280 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:05.369507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.369480 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca" pod "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" (UID: "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:05.369634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.369613 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc" pod "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" (UID: "e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1"). InnerVolumeSpecName "kube-api-access-42pqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:05.468403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.468315 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:05.468403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:05.468348 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pqc\" (UniqueName: \"kubernetes.io/projected/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1-kube-api-access-42pqc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:06.147262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:06.147233 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" Nov 03 03:20:06.147442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:06.147225 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s" event={"ID":"e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1","Type":"ContainerDied","Data":"55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5"} Nov 03 03:20:06.147442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:06.147389 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55485239da38a29fbaac9126f7de078406347f2a116c39a5dbf2f3b3d84c6aa5" Nov 03 03:20:10.068832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.068794 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:20:10.069344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069309 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" containerName="revoke-keys" Nov 03 03:20:10.069344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069321 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" containerName="revoke-keys" Nov 03 03:20:10.069344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069330 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" containerName="revoke-keys" Nov 03 03:20:10.069344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069336 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" containerName="revoke-keys" Nov 03 03:20:10.069586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069473 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" containerName="revoke-keys" Nov 03 03:20:10.069586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.069482 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" containerName="revoke-keys" Nov 03 03:20:10.072918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.072896 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.128003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.127976 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\"" Nov 03 03:20:10.182023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.181986 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:20:10.220926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.220876 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.220982 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221025 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221043 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqztj\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221181 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221201 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221240 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.221375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.221306 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322284 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322333 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322353 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322402 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322436 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322553 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322590 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322624 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322651 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqztj\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322759 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.322929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322808 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.323082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.322932 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.323321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.323122 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.323456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.323436 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.325150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.325121 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.325239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.325220 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.387217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.387181 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqztj\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.387761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.387738 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:10.686926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:10.686814 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:11.187869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:11.184636 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:20:11.200873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:11.189992 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:11.200873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:11.190079 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:11.200873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:11.190119 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:11.217872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:11.215169 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" event={"ID":"c2d5cdee-ba5f-4169-ac86-4a10114549fd","Type":"ContainerStarted","Data":"0bb8b6d0411655167437e2e4ac5e49b22ace6aa8ac583922fb7af3d75c1c5ae4"} Nov 03 03:20:12.223145 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:12.223106 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" event={"ID":"c2d5cdee-ba5f-4169-ac86-4a10114549fd","Type":"ContainerStarted","Data":"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf"} Nov 03 03:20:12.409576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:12.409515 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podStartSLOduration=2.409493837 podStartE2EDuration="2.409493837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:12.406371251 +0000 UTC m=+1103.559059671" watchObservedRunningTime="2026-11-03 03:20:12.409493837 +0000 UTC m=+1103.562182258" Nov 03 03:20:12.687212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:12.687169 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:12.688944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:12.688914 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:12.689099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:12.688963 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:13.687475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:13.687439 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:13.687940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:13.687498 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:14.688133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:14.688101 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:14.688501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:14.688156 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:15.687431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:15.687389 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:15.687627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:15.687454 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:16.687658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:16.687615 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:16.688171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:16.687678 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:17.687395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:17.687354 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:20:17.687623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:17.687424 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 03:20:18.692409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:18.692375 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:18.692933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:18.692673 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:18.693553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:18.693532 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:20:20.426435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.426399 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz"] Nov 03 03:20:20.431519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.431494 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.474664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.474627 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.474896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.474768 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.474968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.474918 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4gl\" (UniqueName: \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.534113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.534072 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:20:20.534113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.534090 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:20:20.534358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.534203 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:20:20.534358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.534297 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:20:20.536814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.536793 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz"] Nov 03 03:20:20.575922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.575890 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.576125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.576006 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4gl\" (UniqueName: \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.576125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.576073 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.576802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.576775 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.579235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.579210 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.637691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.637651 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4gl\" (UniqueName: \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\") pod \"maas-api-revoke-sub-6a1b744a-cmxcz\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:20.745978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:20.745885 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:21.121303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.121270 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz"] Nov 03 03:20:21.122635 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:21.122609 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf78eecf_a94b_41e3_80ce_fb8ee91ef4e9.slice/crio-40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142 WatchSource:0}: Error finding container 40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142: Status 404 returned error can't find the container with id 40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142 Nov 03 03:20:21.293315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.293281 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" event={"ID":"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9","Type":"ContainerStarted","Data":"40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142"} Nov 03 03:20:21.759415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.759326 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:20:21.764151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.764127 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789346 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789567 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkjw\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789605 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789754 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789823 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.789951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.789885 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.790076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.790049 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.790171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.790154 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.790225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.790191 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.816650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.816617 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\"" Nov 03 03:20:21.869176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.869142 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:20:21.891517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.891478 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkjw\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.891704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.891550 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.891704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.891611 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.891704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.891699 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892053 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892121 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892137 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892212 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892316 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892371 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.892813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892782 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.893021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.892999 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.893163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.893140 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.894133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.894101 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.895016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.894994 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.895314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.895289 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.954376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.954059 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkjw\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:21.963222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:21.963194 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:22.084403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.084365 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:22.303388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.302408 5275 generic.go:356] "Generic (PLEG): container finished" podID="af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" containerID="7b41b9d167edc8d5ebe9d67e6ddc6ef88172843454b41f2eec33e6ef270bbb5c" exitCode=0 Nov 03 03:20:22.303388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.302456 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" event={"ID":"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9","Type":"ContainerDied","Data":"7b41b9d167edc8d5ebe9d67e6ddc6ef88172843454b41f2eec33e6ef270bbb5c"} Nov 03 03:20:22.523900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.523857 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:20:22.528258 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:22.528224 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d9fa65_1bbe_44e6_8112_b3e3d7446920.slice/crio-9100a6855388abdcc798d2d874ccf29c4fcac004231cdd396665d4af403ef7e0 WatchSource:0}: Error finding container 9100a6855388abdcc798d2d874ccf29c4fcac004231cdd396665d4af403ef7e0: Status 404 returned error can't find the container with id 9100a6855388abdcc798d2d874ccf29c4fcac004231cdd396665d4af403ef7e0 Nov 03 03:20:22.530892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.530829 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:22.530980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.530948 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:22.531030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:22.530992 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:23.328982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.328938 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" event={"ID":"15d9fa65-1bbe-44e6-8112-b3e3d7446920","Type":"ContainerStarted","Data":"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869"} Nov 03 03:20:23.329469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.329019 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" event={"ID":"15d9fa65-1bbe-44e6-8112-b3e3d7446920","Type":"ContainerStarted","Data":"9100a6855388abdcc798d2d874ccf29c4fcac004231cdd396665d4af403ef7e0"} Nov 03 03:20:23.513953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.513743 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podStartSLOduration=2.5137211600000002 podStartE2EDuration="2.51372116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:23.510260295 +0000 UTC m=+1114.662948753" watchObservedRunningTime="2026-11-03 03:20:23.51372116 +0000 UTC m=+1114.666409585" Nov 03 03:20:23.623251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.623224 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:23.755878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.750481 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\") pod \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " Nov 03 03:20:23.755878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.750539 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\" (UniqueName: \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\") pod \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " Nov 03 03:20:23.755878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.750771 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\") pod \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\" (UID: \"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9\") " Nov 03 03:20:23.755878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.751586 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca" pod "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" (UID: "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:23.792974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.792355 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca" pod "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" (UID: "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:23.795840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.795794 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl" pod "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" (UID: "af78eecf-a94b-41e3-80ce-fb8ee91ef4e9"). InnerVolumeSpecName "kube-api-access-dx4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:23.859937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.859699 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:23.859937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.859743 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:23.859937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:23.859761 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4gl\" (UniqueName: \"kubernetes.io/projected/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9-kube-api-access-dx4gl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:24.086900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.085226 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:24.098541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.088033 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:24.098541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.088093 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:24.347001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.342361 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" Nov 03 03:20:24.350869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.347904 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz" event={"ID":"af78eecf-a94b-41e3-80ce-fb8ee91ef4e9","Type":"ContainerDied","Data":"40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142"} Nov 03 03:20:24.350869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:24.347966 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d4284ff400dee7e95b3fb07fba179834e07201efb78059a95b9fc718716142" Nov 03 03:20:25.088943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:25.087250 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:25.088943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:25.087363 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:26.095326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:26.095291 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:26.096009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:26.095977 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:27.085666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:27.085618 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:27.085833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:27.085688 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:28.085457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:28.085424 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:28.085875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:28.085478 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:29.085251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:29.085209 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:20:29.085436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:29.085274 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 03:20:30.090134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:30.090101 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:30.090612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:30.090597 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:30.091454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:30.091429 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:20:31.247117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.247075 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr"] Nov 03 03:20:31.247916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.247892 5275 cpu_manager.go:414] "containerMap: removing container" podUID="af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" containerName="revoke-keys" Nov 03 03:20:31.248039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.247918 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" containerName="revoke-keys" Nov 03 03:20:31.248101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.248063 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" containerName="revoke-keys" Nov 03 03:20:31.252744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.252720 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.280571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.280522 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnm5p\" (UniqueName: \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.280763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.280606 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.280763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.280647 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.305304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.305270 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:20:31.306001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.305937 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:20:31.309700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.309675 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr"] Nov 03 03:20:31.353141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.353099 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:20:31.353792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.353765 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:20:31.382201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.382155 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnm5p\" (UniqueName: \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.382403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.382265 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.382403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.382313 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.383332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.383302 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.385001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.384972 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.485224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.485171 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnm5p\" (UniqueName: \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\") pod \"maas-api-revoke-sub-7ce39bde-2s6qr\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:31.578869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:31.578560 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:32.024492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:32.021318 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr"] Nov 03 03:20:32.340920 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:20:32.340875 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:20:32.483262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:32.483208 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" event={"ID":"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc","Type":"ContainerStarted","Data":"f896ec48f5dda965fb89ac929dd3166b96ad7b7f877f1913c3f6cb457b7e9ebc"} Nov 03 03:20:33.500118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:33.500006 5275 generic.go:356] "Generic (PLEG): container finished" podID="9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" containerID="70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07" exitCode=0 Nov 03 03:20:33.500588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:33.500175 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" event={"ID":"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc","Type":"ContainerDied","Data":"70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07"} Nov 03 03:20:34.732071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.732041 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:34.834337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.834298 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\" (UniqueName: \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\") pod \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " Nov 03 03:20:34.834528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.834382 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\") pod \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " Nov 03 03:20:34.834528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.834444 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\") pod \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\" (UID: \"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc\") " Nov 03 03:20:34.834838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.834809 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca" pod "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" (UID: "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:34.836646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.836622 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca" pod "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" (UID: "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:34.836718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.836624 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p" pod "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" (UID: "9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc"). InnerVolumeSpecName "kube-api-access-cnm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:34.935709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.935664 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:34.935709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.935701 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnm5p\" (UniqueName: \"kubernetes.io/projected/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-kube-api-access-cnm5p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:34.935709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:34.935715 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:35.521325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:35.521294 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" Nov 03 03:20:35.521533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:35.521312 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr" event={"ID":"9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc","Type":"ContainerDied","Data":"f896ec48f5dda965fb89ac929dd3166b96ad7b7f877f1913c3f6cb457b7e9ebc"} Nov 03 03:20:35.521533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:35.521368 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f896ec48f5dda965fb89ac929dd3166b96ad7b7f877f1913c3f6cb457b7e9ebc" Nov 03 03:20:48.036170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.036138 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb"] Nov 03 03:20:48.036665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.036651 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" containerName="revoke-keys" Nov 03 03:20:48.036665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.036665 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" containerName="revoke-keys" Nov 03 03:20:48.036788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.036753 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" containerName="revoke-keys" Nov 03 03:20:48.041332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.041308 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.099128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.099072 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb"] Nov 03 03:20:48.144567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.144533 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:20:48.144567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.144533 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:20:48.144770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.144537 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:20:48.144770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.144633 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:20:48.197048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.197008 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rs58\" (UniqueName: \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.197244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.197158 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.197334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.197306 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.299078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.298987 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rs58\" (UniqueName: \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.299258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.299142 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.299258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.299219 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.299959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.299933 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.302053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.302026 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.360717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.360677 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rs58\" (UniqueName: \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\") pod \"maas-api-revoke-sub-238c6a5f-99vsb\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:48.655393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:48.655308 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:49.036955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.036927 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb"] Nov 03 03:20:49.039297 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:49.039270 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7d66c_ead1_4392_8295_6bde48cfaf7a.slice/crio-dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc WatchSource:0}: Error finding container dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc: Status 404 returned error can't find the container with id dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc Nov 03 03:20:49.403184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.402701 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:20:49.407787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.407757 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.497152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.497114 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:20:49.512386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512357 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512459 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512487 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8w64\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512535 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512567 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512601 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512624 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512697 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.512922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.512748 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630049 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630113 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630166 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630201 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630291 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630359 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630391 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630481 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.630700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.630507 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8w64\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.638892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.631294 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.644542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.639549 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.644542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.639865 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.645275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.645073 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.646149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.646109 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.646458 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.646439 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.671879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.654438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.673869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.673714 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" event={"ID":"78e7d66c-ead1-4392-8295-6bde48cfaf7a","Type":"ContainerStarted","Data":"3eb4e593ae091070482a9a56cc29f98803e99558f43a9090c0ea85a83572e32c"} Nov 03 03:20:49.673869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.673779 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" event={"ID":"78e7d66c-ead1-4392-8295-6bde48cfaf7a","Type":"ContainerStarted","Data":"dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc"} Nov 03 03:20:49.692760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.692724 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8w64\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.700559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.700527 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-skdrn\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.726624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.726590 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:49.794772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:49.794724 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" podStartSLOduration=2.794707593 podStartE2EDuration="2.794707593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:49.791748027 +0000 UTC m=+1140.944436452" watchObservedRunningTime="2026-11-03 03:20:49.794707593 +0000 UTC m=+1140.947396013" Nov 03 03:20:50.168331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.168299 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:20:50.170698 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:20:50.170669 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce8feec_eabe_42cc_8638_07e5ab35da90.slice/crio-aeb7617b61eb9a7b611b95df5b111a42206245c33de0f750cf6426ea39ee450e WatchSource:0}: Error finding container aeb7617b61eb9a7b611b95df5b111a42206245c33de0f750cf6426ea39ee450e: Status 404 returned error can't find the container with id aeb7617b61eb9a7b611b95df5b111a42206245c33de0f750cf6426ea39ee450e Nov 03 03:20:50.173649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.173607 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:50.173747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.173702 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:50.173835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.173742 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:20:50.684449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.684357 5275 generic.go:356] "Generic (PLEG): container finished" podID="78e7d66c-ead1-4392-8295-6bde48cfaf7a" containerID="3eb4e593ae091070482a9a56cc29f98803e99558f43a9090c0ea85a83572e32c" exitCode=0 Nov 03 03:20:50.684617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.684513 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" event={"ID":"78e7d66c-ead1-4392-8295-6bde48cfaf7a","Type":"ContainerDied","Data":"3eb4e593ae091070482a9a56cc29f98803e99558f43a9090c0ea85a83572e32c"} Nov 03 03:20:50.686559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.686529 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" event={"ID":"7ce8feec-eabe-42cc-8638-07e5ab35da90","Type":"ContainerStarted","Data":"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2"} Nov 03 03:20:50.686689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.686589 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" event={"ID":"7ce8feec-eabe-42cc-8638-07e5ab35da90","Type":"ContainerStarted","Data":"aeb7617b61eb9a7b611b95df5b111a42206245c33de0f750cf6426ea39ee450e"} Nov 03 03:20:50.727141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.727097 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:50.929038 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:50.928960 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podStartSLOduration=1.928936413 podStartE2EDuration="1.928936413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:50.923719527 +0000 UTC m=+1142.076407974" watchObservedRunningTime="2026-11-03 03:20:50.928936413 +0000 UTC m=+1142.081624837" Nov 03 03:20:51.730303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:51.729924 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:51.730303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:51.729989 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:52.384733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.384375 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:52.554968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.553779 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\") pod \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " Nov 03 03:20:52.554968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.553941 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\" (UniqueName: \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\") pod \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " Nov 03 03:20:52.554968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.554039 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\") pod \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\" (UID: \"78e7d66c-ead1-4392-8295-6bde48cfaf7a\") " Nov 03 03:20:52.647400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.647067 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca" pod "78e7d66c-ead1-4392-8295-6bde48cfaf7a" (UID: "78e7d66c-ead1-4392-8295-6bde48cfaf7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:20:52.684556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.683793 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:52.690616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.690543 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58" pod "78e7d66c-ead1-4392-8295-6bde48cfaf7a" (UID: "78e7d66c-ead1-4392-8295-6bde48cfaf7a"). InnerVolumeSpecName "kube-api-access-6rs58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:52.724168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.702022 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca" pod "78e7d66c-ead1-4392-8295-6bde48cfaf7a" (UID: "78e7d66c-ead1-4392-8295-6bde48cfaf7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:52.727933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.727333 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:52.727933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.727390 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:52.735912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.734059 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" Nov 03 03:20:52.741438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.741114 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb" event={"ID":"78e7d66c-ead1-4392-8295-6bde48cfaf7a","Type":"ContainerDied","Data":"dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc"} Nov 03 03:20:52.741438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.741176 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc903ed5dde48d67397093bbb161eb4de4d8ea40c52a13997fdcad4df4ae3bc" Nov 03 03:20:52.786430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.786391 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e7d66c-ead1-4392-8295-6bde48cfaf7a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:52.787652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:52.787625 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rs58\" (UniqueName: \"kubernetes.io/projected/78e7d66c-ead1-4392-8295-6bde48cfaf7a-kube-api-access-6rs58\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:20:52.870934 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:20:52.869976 5275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7d66c_ead1_4392_8295_6bde48cfaf7a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 03:20:53.727770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:53.727732 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:53.727982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:53.727785 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:54.728092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:54.728054 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:54.728494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:54.728111 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:55.727342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:55.727299 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:55.727556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:55.727360 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:56.727612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:56.727578 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:56.728083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:56.727644 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:57.727769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:57.727729 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 03:20:57.728202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:57.727796 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 03:20:59.101664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:59.101632 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:59.102219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:59.101960 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:20:59.102758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:20:59.102732 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:21:02.330645 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:21:02.330608 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:21:04.518650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.518608 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:21:04.519225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.519207 5275 cpu_manager.go:414] "containerMap: removing container" podUID="78e7d66c-ead1-4392-8295-6bde48cfaf7a" containerName="revoke-keys" Nov 03 03:21:04.519272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.519226 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e7d66c-ead1-4392-8295-6bde48cfaf7a" containerName="revoke-keys" Nov 03 03:21:04.519343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.519334 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e7d66c-ead1-4392-8295-6bde48cfaf7a" containerName="revoke-keys" Nov 03 03:21:04.523036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.523016 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.620863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.620809 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:21:04.630783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.630753 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.630995 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.630813 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.630995 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.630951 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631004 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631113 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631154 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpk24\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631237 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631275 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.631310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.631296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.732726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732695 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.732726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732729 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732794 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732835 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732925 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.732969 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733014 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733048 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpk24\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733260 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733291 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733338 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733423 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733657 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.733936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.733914 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.735444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.735416 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.735541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.735526 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.793879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.793818 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpk24\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.794042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.793823 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6948b\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:04.836322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:04.836279 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:05.271281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.271249 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:21:05.272618 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:21:05.272597 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ef7623_310e_4f3b_b24a_22d1410ba547.slice/crio-d4d715d5fe23725e7c974234abfa8f32082fa6791ce2f80cffe7e558b1273bb3 WatchSource:0}: Error finding container d4d715d5fe23725e7c974234abfa8f32082fa6791ce2f80cffe7e558b1273bb3: Status 404 returned error can't find the container with id d4d715d5fe23725e7c974234abfa8f32082fa6791ce2f80cffe7e558b1273bb3 Nov 03 03:21:05.275611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.275580 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:05.275691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.275642 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:05.275691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.275670 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:05.860871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.860069 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" event={"ID":"36ef7623-310e-4f3b-b24a-22d1410ba547","Type":"ContainerStarted","Data":"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b"} Nov 03 03:21:05.860871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:05.860133 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" event={"ID":"36ef7623-310e-4f3b-b24a-22d1410ba547","Type":"ContainerStarted","Data":"d4d715d5fe23725e7c974234abfa8f32082fa6791ce2f80cffe7e558b1273bb3"} Nov 03 03:21:06.838088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:06.838053 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:06.842477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:06.842442 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:06.842745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:06.842718 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:07.837286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:07.837242 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:07.837697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:07.837319 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:08.846874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:08.839786 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:08.846874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:08.840044 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:09.839871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:09.839395 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:09.839871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:09.839476 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:10.211929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.204610 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podStartSLOduration=6.204591518 podStartE2EDuration="6.204591518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:05.98420366 +0000 UTC m=+1157.136892081" watchObservedRunningTime="2026-11-03 03:21:10.204591518 +0000 UTC m=+1161.357279949" Nov 03 03:21:10.221873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.221414 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:21:10.226885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.226436 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.280927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.280876 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\"" Nov 03 03:21:10.338531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.338472 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:21:10.407915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.407833 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.407921 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.407948 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gjf\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408019 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408080 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408215 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408277 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408310 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.408380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.408340 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509333 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509385 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509409 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509426 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509495 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509522 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509539 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gjf\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509591 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.509744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.509606 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.510068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.510005 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.510472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.510236 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.510472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.510434 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.510623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.510505 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.510623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.510513 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.512932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.512897 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.513082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.513059 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.581088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.580532 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gjf\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.581088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.580999 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:10.837648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.837618 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:10.837833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.837670 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:10.869874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:10.869030 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:11.309809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.309767 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:21:11.344206 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:21:11.344175 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fd9bba_181d_4e48_bb2d_1585b2c39258.slice/crio-568c0b6da3b65eb426ba15f7f06c34439a84e2aa9a0ee615d378675556aec95b WatchSource:0}: Error finding container 568c0b6da3b65eb426ba15f7f06c34439a84e2aa9a0ee615d378675556aec95b: Status 404 returned error can't find the container with id 568c0b6da3b65eb426ba15f7f06c34439a84e2aa9a0ee615d378675556aec95b Nov 03 03:21:11.347395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.347358 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:11.347511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.347446 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:11.347511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.347493 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:21:11.836954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.836913 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:11.837133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.836972 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:11.994493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.994451 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" event={"ID":"f9fd9bba-181d-4e48-bb2d-1585b2c39258","Type":"ContainerStarted","Data":"1c0d9984c8435d90364602fdb8b4642d30f2d05b07a404e454e6bc353525fb4e"} Nov 03 03:21:11.994672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:11.994504 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" event={"ID":"f9fd9bba-181d-4e48-bb2d-1585b2c39258","Type":"ContainerStarted","Data":"568c0b6da3b65eb426ba15f7f06c34439a84e2aa9a0ee615d378675556aec95b"} Nov 03 03:21:12.121202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.121075 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podStartSLOduration=2.121059495 podStartE2EDuration="2.121059495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:12.118580595 +0000 UTC m=+1163.271269019" watchObservedRunningTime="2026-11-03 03:21:12.121059495 +0000 UTC m=+1163.273747916" Nov 03 03:21:12.836985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.836948 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 03:21:12.837541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.837003 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 03:21:12.875873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.874950 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:12.875873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.875085 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:12.875873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:12.875127 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:13.879282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:13.879245 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:13.879972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:13.879939 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:14.837554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:14.837520 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:14.837772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:14.837591 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:14.874251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:14.874218 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:14.874486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:14.874457 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:15.838261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:15.838225 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:15.838732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:15.838291 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:15.879876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:15.877979 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:15.879876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:15.878056 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:16.875307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:16.875266 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:16.876039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:16.876000 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:16.880883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:16.880032 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:16.880883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:16.880084 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:17.556270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:17.549888 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:17.556270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:17.550372 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:17.556270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:17.551197 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:21:17.872003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:17.871913 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 03:21:17.872003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:17.871980 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 03:21:18.875503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:18.875474 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:18.876030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:18.875949 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:18.876651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:18.876627 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:21:23.543154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:23.536046 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:23.543154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:23.536148 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:24.037871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:24.032975 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:24.166004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:24.165962 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:21:24.258873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:24.257919 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:21:32.330645 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:21:32.330606 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:21:49.158979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:49.158947 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:21:49.163484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:49.163462 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:21:54.375499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.375464 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:21:54.381668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.381640 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.382755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.382728 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.382902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.382814 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.382973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.382948 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhlm\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.383156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.383136 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.483120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.483086 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b44120f006ba\"" Nov 03 03:21:54.483443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.483424 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1db6abbb545c\"" Nov 03 03:21:54.483617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.483603 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 03:21:54.483826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.483784 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 03:21:54.484006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.483992 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 03:21:54.484181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.484166 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 03:21:54.484392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.484367 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6867f294d875\"" Nov 03 03:21:54.484508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.484424 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6czs8\"" Nov 03 03:21:54.485144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.485121 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.485241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.485205 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.485313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.485266 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.485433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.485339 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhlm\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.486710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.486687 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.487730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.487709 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.489045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.489026 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:21:54.489247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.489223 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.546176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.546144 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhlm\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\") pod \"payload-processing-external-model-866d94f86b-2kpvr\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:54.697633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:54.697544 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:21:55.077963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:55.077929 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:21:55.079421 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:21:55.079396 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1a4a81_6298_448f_ad39_1f770dfcc4ac.slice/crio-557b2c4b58028e4c382f3ba3616516c3595feb83107e82315d85099a502d05f4 WatchSource:0}: Error finding container 557b2c4b58028e4c382f3ba3616516c3595feb83107e82315d85099a502d05f4: Status 404 returned error can't find the container with id 557b2c4b58028e4c382f3ba3616516c3595feb83107e82315d85099a502d05f4 Nov 03 03:21:55.375920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:55.375789 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" event={"ID":"8d1a4a81-6298-448f-ad39-1f770dfcc4ac","Type":"ContainerStarted","Data":"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e"} Nov 03 03:21:55.375920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:55.375875 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" event={"ID":"8d1a4a81-6298-448f-ad39-1f770dfcc4ac","Type":"ContainerStarted","Data":"557b2c4b58028e4c382f3ba3616516c3595feb83107e82315d85099a502d05f4"} Nov 03 03:21:55.376418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:21:55.376077 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:22:02.329895 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:22:02.329842 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:22:06.392477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:06.392440 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:22:06.521993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:06.521918 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" podStartSLOduration=12.521895015 podStartE2EDuration="12.521895015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:55.493971677 +0000 UTC m=+1206.646660100" watchObservedRunningTime="2026-11-03 03:22:06.521895015 +0000 UTC m=+1217.674583436" Nov 03 03:22:20.025874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:20.024934 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:22:20.047006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:20.046952 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:22:32.330012 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:22:32.329972 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:22:49.190593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:49.190561 5275 scope.go:122] "RemoveContainer" containerID="56ac9c337bed0d8c01472831ad92b5ced37736b7328b055fc2dfd8079ac54dd6" Nov 03 03:22:49.201004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:49.200976 5275 scope.go:122] "RemoveContainer" containerID="f6adc05149d7a7ddd16ce1ff8d39c55eba8413ad654f4c3f9f527e23d0d9951f" Nov 03 03:22:49.210956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:49.210932 5275 scope.go:122] "RemoveContainer" containerID="1bd66bd2284bedb6e4b6bfb9728649ccd10ec89b203685b8a0cba7ece05112c0" Nov 03 03:22:49.221738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:49.221711 5275 scope.go:122] "RemoveContainer" containerID="e5cb8a46da01534f0820d92b9bce1ca7da28d139620922af8bef5f81f79efbee" Nov 03 03:22:49.232486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:49.232462 5275 scope.go:122] "RemoveContainer" containerID="0bcc8d8a5d42250e527fbef907bbc188cc1274d4952faf592fe3f9d696c13558" Nov 03 03:22:56.376956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.376861 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:22:56.383321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.383292 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.451584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.451541 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.451804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.451597 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklmx\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.451804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.451699 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.451804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.451749 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.452037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.451906 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.452037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.452011 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.452161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.452103 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.452222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.452161 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.452281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.452243 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.502581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.494369 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:22:56.553809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.553767 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.553823 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.553893 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.553939 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.553983 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554008 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklmx\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554042 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554086 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554158 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554311 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554504 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554768 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.554796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.554781 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.555054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.555033 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.557365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.557334 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.557617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.557594 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.616388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.616357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklmx\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.616574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.616357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:56.703806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:56.703695 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:57.144751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:57.144721 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:22:57.146200 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:22:57.146174 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944539d9_6813_4673_919b_e896606acb5e.slice/crio-bfe2eb93180dc4f958ed6366805557a92bf228806fbf2c93a51c98a6ca650812 WatchSource:0}: Error finding container bfe2eb93180dc4f958ed6366805557a92bf228806fbf2c93a51c98a6ca650812: Status 404 returned error can't find the container with id bfe2eb93180dc4f958ed6366805557a92bf228806fbf2c93a51c98a6ca650812 Nov 03 03:22:57.148451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:57.148420 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:22:57.148536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:57.148487 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:22:57.148536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:57.148518 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:22:58.114647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.114615 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" event={"ID":"944539d9-6813-4673-919b-e896606acb5e","Type":"ContainerStarted","Data":"f33c773509d37523543a81c6b3bd33f36f297e68f90ee972f84321d30e72c561"} Nov 03 03:22:58.115182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.114672 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" event={"ID":"944539d9-6813-4673-919b-e896606acb5e","Type":"ContainerStarted","Data":"bfe2eb93180dc4f958ed6366805557a92bf228806fbf2c93a51c98a6ca650812"} Nov 03 03:22:58.288870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.288369 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podStartSLOduration=2.288350469 podStartE2EDuration="2.288350469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:58.286164277 +0000 UTC m=+1269.438852739" watchObservedRunningTime="2026-11-03 03:22:58.288350469 +0000 UTC m=+1269.441038883" Nov 03 03:22:58.704380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.704339 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:22:58.706423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.706387 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:22:58.706591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:58.706454 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:22:59.704596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:59.704566 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:22:59.705036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:22:59.704632 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:23:00.704930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:00.704888 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:23:00.705547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:00.704959 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:23:01.704184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:01.704150 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:23:01.704375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:01.704207 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:23:02.330917 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:23:02.330873 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:23:02.704899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:02.704760 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:23:02.704899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:02.704822 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:23:03.704112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:03.704073 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 03:23:03.704703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:03.704142 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 03:23:04.708411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:04.708377 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:23:04.708916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:04.708740 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:23:04.709481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:04.709461 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:23:19.899352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.899307 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:23:19.903442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.903417 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952421 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952568 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952620 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbp9\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952725 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952805 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952900 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.952970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952958 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:19.953006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:19.952977 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.010933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.010898 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:23:20.054258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054208 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054255 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054529 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054366 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054584 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054625 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbp9\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054673 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054751 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054816 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.054927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.054907 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.055069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.055049 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.055256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.055233 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.055331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.055303 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.055392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.055373 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.056001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.055982 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.057399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.057379 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.057936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.057916 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.115672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.115641 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.115834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.115725 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbp9\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.218317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.218225 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:20.668709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.668684 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:23:20.669502 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:23:20.669480 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac85226_99d9_4c5f_838f_3e51581078b5.slice/crio-8e0d9423efc39a47438797bb06be9488658026bc4917c4795baf2cc2462221c3 WatchSource:0}: Error finding container 8e0d9423efc39a47438797bb06be9488658026bc4917c4795baf2cc2462221c3: Status 404 returned error can't find the container with id 8e0d9423efc39a47438797bb06be9488658026bc4917c4795baf2cc2462221c3 Nov 03 03:23:20.671746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.671711 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:23:20.671857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.671779 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:23:20.671857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:20.671807 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:23:21.300982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:21.300940 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" event={"ID":"7ac85226-99d9-4c5f-838f-3e51581078b5","Type":"ContainerStarted","Data":"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c"} Nov 03 03:23:21.301389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:21.301001 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" event={"ID":"7ac85226-99d9-4c5f-838f-3e51581078b5","Type":"ContainerStarted","Data":"8e0d9423efc39a47438797bb06be9488658026bc4917c4795baf2cc2462221c3"} Nov 03 03:23:21.442057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:21.442005 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podStartSLOduration=2.441990427 podStartE2EDuration="2.441990427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:21.437515172 +0000 UTC m=+1292.590203594" watchObservedRunningTime="2026-11-03 03:23:21.441990427 +0000 UTC m=+1292.594678848" Nov 03 03:23:22.218865 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:22.218826 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:22.220730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:22.220703 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:22.220878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:22.220750 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:23.219204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:23.219168 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:23.219586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:23.219221 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:24.219534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:24.219488 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:24.220098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:24.219556 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:25.218906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:25.218842 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:25.219090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:25.218935 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:26.219281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:26.219241 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:26.219732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:26.219311 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:27.218899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:27.218824 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:23:27.219161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:27.218924 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 03:23:28.223573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:28.223539 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:28.224111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:28.223950 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:28.224671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:28.224652 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:23:32.330528 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:23:32.330489 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:23:51.670243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.670202 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb"] Nov 03 03:23:51.674156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.674131 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.727533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.727500 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:23:51.732228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.732198 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb"] Nov 03 03:23:51.755799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.755759 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.756061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.756012 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtb9\" (UniqueName: \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.756140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.756069 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.777364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.777330 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:23:51.777548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.777383 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:23:51.827879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.827819 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:23:51.857706 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.857659 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.857937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.857914 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.858067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.858047 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtb9\" (UniqueName: \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.858669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.858643 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.860269 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.860239 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.919732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.919697 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtb9\" (UniqueName: \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\") pod \"maas-api-revoke-sub-0c532295-tzbdb\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:51.989075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:51.988992 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:52.408198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:52.408164 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb"] Nov 03 03:23:52.409528 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:23:52.409496 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41102a61_589b_4bea_85fb_3cf5e6f64a4f.slice/crio-805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250 WatchSource:0}: Error finding container 805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250: Status 404 returned error can't find the container with id 805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250 Nov 03 03:23:52.525131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:52.525093 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" event={"ID":"41102a61-589b-4bea-85fb-3cf5e6f64a4f","Type":"ContainerStarted","Data":"805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250"} Nov 03 03:23:53.533579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:53.533542 5275 generic.go:356] "Generic (PLEG): container finished" podID="41102a61-589b-4bea-85fb-3cf5e6f64a4f" containerID="004ee378ee55a01f9f44f7781e2da2ab57e9eba156b7b2e6837aae91b417aaa1" exitCode=0 Nov 03 03:23:53.533975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:53.533621 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" event={"ID":"41102a61-589b-4bea-85fb-3cf5e6f64a4f","Type":"ContainerDied","Data":"004ee378ee55a01f9f44f7781e2da2ab57e9eba156b7b2e6837aae91b417aaa1"} Nov 03 03:23:54.718628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.718601 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:54.810651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.810562 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\" (UniqueName: \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\") pod \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " Nov 03 03:23:54.810651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.810615 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\") pod \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " Nov 03 03:23:54.810837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.810685 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\") pod \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\" (UID: \"41102a61-589b-4bea-85fb-3cf5e6f64a4f\") " Nov 03 03:23:54.811103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.811064 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca" pod "41102a61-589b-4bea-85fb-3cf5e6f64a4f" (UID: "41102a61-589b-4bea-85fb-3cf5e6f64a4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:23:54.811237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.811195 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:54.812959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.812925 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca" pod "41102a61-589b-4bea-85fb-3cf5e6f64a4f" (UID: "41102a61-589b-4bea-85fb-3cf5e6f64a4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:23:54.813059 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.812975 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9" pod "41102a61-589b-4bea-85fb-3cf5e6f64a4f" (UID: "41102a61-589b-4bea-85fb-3cf5e6f64a4f"). InnerVolumeSpecName "kube-api-access-8dtb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:23:54.912612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.912574 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtb9\" (UniqueName: \"kubernetes.io/projected/41102a61-589b-4bea-85fb-3cf5e6f64a4f-kube-api-access-8dtb9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:54.912612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:54.912601 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41102a61-589b-4bea-85fb-3cf5e6f64a4f-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:55.548217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:55.548179 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" Nov 03 03:23:55.548391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:55.548216 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb" event={"ID":"41102a61-589b-4bea-85fb-3cf5e6f64a4f","Type":"ContainerDied","Data":"805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250"} Nov 03 03:23:55.548391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:23:55.548269 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805235b5ff7711ca7aee2ed6fb7b43b8abdd12f558b9120904cc7f670ab77250" Nov 03 03:24:00.061039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:00.060988 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc"] Nov 03 03:24:00.108789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:00.108752 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ad2098-wfwlc"] Nov 03 03:24:00.985049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:00.985010 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e98d58-f759-4dd3-89d2-4b79b75d7914" path="/var/lib/kubelet/pods/31e98d58-f759-4dd3-89d2-4b79b75d7914/volumes" Nov 03 03:24:02.331393 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:24:02.331352 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:24:04.006766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.006724 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59"] Nov 03 03:24:04.007554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.007536 5275 cpu_manager.go:414] "containerMap: removing container" podUID="41102a61-589b-4bea-85fb-3cf5e6f64a4f" containerName="revoke-keys" Nov 03 03:24:04.007615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.007556 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41102a61-589b-4bea-85fb-3cf5e6f64a4f" containerName="revoke-keys" Nov 03 03:24:04.007689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.007676 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="41102a61-589b-4bea-85fb-3cf5e6f64a4f" containerName="revoke-keys" Nov 03 03:24:04.014551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.012224 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.067024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.066992 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59"] Nov 03 03:24:04.112007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.111974 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:04.112179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.112017 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:04.112179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.112095 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:04.114036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.114020 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:04.123332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.123303 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.123400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.123343 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xrq\" (UniqueName: \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.123536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.123522 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.224639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.224604 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.224829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.224725 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.224829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.224747 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xrq\" (UniqueName: \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.225526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.225501 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.227198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.227178 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.285619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.285590 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xrq\" (UniqueName: \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\") pod \"maas-api-revoke-sub-bfd5e5d7-cnn59\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.335990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.335945 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:04.723611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:04.723583 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59"] Nov 03 03:24:04.724991 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:04.724964 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fa4a6c_d363_438c_839c_3a155a5ae6a8.slice/crio-68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f WatchSource:0}: Error finding container 68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f: Status 404 returned error can't find the container with id 68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f Nov 03 03:24:05.631920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:05.631881 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" event={"ID":"b7fa4a6c-d363-438c-839c-3a155a5ae6a8","Type":"ContainerStarted","Data":"5a0a282998725c65dcdfb479e686ef26520b8acee8f3ae9ec90b27d62ab6114d"} Nov 03 03:24:05.632342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:05.631946 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" event={"ID":"b7fa4a6c-d363-438c-839c-3a155a5ae6a8","Type":"ContainerStarted","Data":"68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f"} Nov 03 03:24:05.749137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:05.749085 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" podStartSLOduration=2.749071343 podStartE2EDuration="2.749071343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:05.746869235 +0000 UTC m=+1336.899557654" watchObservedRunningTime="2026-11-03 03:24:05.749071343 +0000 UTC m=+1336.901759763" Nov 03 03:24:06.641149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:06.641116 5275 generic.go:356] "Generic (PLEG): container finished" podID="b7fa4a6c-d363-438c-839c-3a155a5ae6a8" containerID="5a0a282998725c65dcdfb479e686ef26520b8acee8f3ae9ec90b27d62ab6114d" exitCode=0 Nov 03 03:24:06.641592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:06.641156 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" event={"ID":"b7fa4a6c-d363-438c-839c-3a155a5ae6a8","Type":"ContainerDied","Data":"5a0a282998725c65dcdfb479e686ef26520b8acee8f3ae9ec90b27d62ab6114d"} Nov 03 03:24:07.801652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.801623 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:07.970499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.970413 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\") pod \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " Nov 03 03:24:07.970672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.970522 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\" (UniqueName: \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\") pod \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " Nov 03 03:24:07.970672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.970591 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\") pod \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\" (UID: \"b7fa4a6c-d363-438c-839c-3a155a5ae6a8\") " Nov 03 03:24:07.970876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.970824 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca" pod "b7fa4a6c-d363-438c-839c-3a155a5ae6a8" (UID: "b7fa4a6c-d363-438c-839c-3a155a5ae6a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:07.971214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.971177 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:07.973402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.973368 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca" pod "b7fa4a6c-d363-438c-839c-3a155a5ae6a8" (UID: "b7fa4a6c-d363-438c-839c-3a155a5ae6a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:07.973489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:07.973399 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq" pod "b7fa4a6c-d363-438c-839c-3a155a5ae6a8" (UID: "b7fa4a6c-d363-438c-839c-3a155a5ae6a8"). InnerVolumeSpecName "kube-api-access-b5xrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:08.072748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.072710 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xrq\" (UniqueName: \"kubernetes.io/projected/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-kube-api-access-b5xrq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:08.072748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.072740 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7fa4a6c-d363-438c-839c-3a155a5ae6a8-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:08.293012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.292974 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46"] Nov 03 03:24:08.293532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.293518 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b7fa4a6c-d363-438c-839c-3a155a5ae6a8" containerName="revoke-keys" Nov 03 03:24:08.293587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.293534 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fa4a6c-d363-438c-839c-3a155a5ae6a8" containerName="revoke-keys" Nov 03 03:24:08.293637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.293628 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fa4a6c-d363-438c-839c-3a155a5ae6a8" containerName="revoke-keys" Nov 03 03:24:08.297157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.297139 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.355904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.355871 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46"] Nov 03 03:24:08.478588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.478543 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.478875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.478836 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.478959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.478892 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtln\" (UniqueName: \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.580598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.580506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.580598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.580555 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtln\" (UniqueName: \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.580837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.580630 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.581294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.581273 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.583129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.583105 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.641948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.641918 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtln\" (UniqueName: \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\") pod \"maas-api-revoke-sub-c26a8da5-cmn46\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:08.655589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.655560 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" Nov 03 03:24:08.655589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.655575 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59" event={"ID":"b7fa4a6c-d363-438c-839c-3a155a5ae6a8","Type":"ContainerDied","Data":"68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f"} Nov 03 03:24:08.655773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.655645 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b75128a5052e52c825870122d3104c447decc9384e21530dda366d7e36e03f" Nov 03 03:24:08.911183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:08.911096 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:09.291738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:09.291705 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46"] Nov 03 03:24:09.292410 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:09.292383 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb5e3be_84fc_41e3_adec_0848199a73cb.slice/crio-a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0 WatchSource:0}: Error finding container a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0: Status 404 returned error can't find the container with id a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0 Nov 03 03:24:09.665072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:09.665031 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" event={"ID":"bfb5e3be-84fc-41e3-adec-0848199a73cb","Type":"ContainerStarted","Data":"1ab578d79c9b94fb945ab741109804dd98f7ed0ae934db2e0d4165ec7260120a"} Nov 03 03:24:09.665259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:09.665097 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" event={"ID":"bfb5e3be-84fc-41e3-adec-0848199a73cb","Type":"ContainerStarted","Data":"a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0"} Nov 03 03:24:09.783088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:09.783029 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" podStartSLOduration=1.783014675 podStartE2EDuration="1.783014675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:09.779898555 +0000 UTC m=+1340.932586977" watchObservedRunningTime="2026-11-03 03:24:09.783014675 +0000 UTC m=+1340.935703097" Nov 03 03:24:10.675873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:10.675047 5275 generic.go:356] "Generic (PLEG): container finished" podID="bfb5e3be-84fc-41e3-adec-0848199a73cb" containerID="1ab578d79c9b94fb945ab741109804dd98f7ed0ae934db2e0d4165ec7260120a" exitCode=0 Nov 03 03:24:10.675873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:10.675260 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" event={"ID":"bfb5e3be-84fc-41e3-adec-0848199a73cb","Type":"ContainerDied","Data":"1ab578d79c9b94fb945ab741109804dd98f7ed0ae934db2e0d4165ec7260120a"} Nov 03 03:24:11.834931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.833580 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:11.927167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.927133 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\" (UniqueName: \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\") pod \"bfb5e3be-84fc-41e3-adec-0848199a73cb\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " Nov 03 03:24:11.927338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.927320 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\") pod \"bfb5e3be-84fc-41e3-adec-0848199a73cb\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " Nov 03 03:24:11.927422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.927348 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\") pod \"bfb5e3be-84fc-41e3-adec-0848199a73cb\" (UID: \"bfb5e3be-84fc-41e3-adec-0848199a73cb\") " Nov 03 03:24:11.927774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.927747 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca" pod "bfb5e3be-84fc-41e3-adec-0848199a73cb" (UID: "bfb5e3be-84fc-41e3-adec-0848199a73cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:11.929413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.929385 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln" pod "bfb5e3be-84fc-41e3-adec-0848199a73cb" (UID: "bfb5e3be-84fc-41e3-adec-0848199a73cb"). InnerVolumeSpecName "kube-api-access-gxtln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:11.929541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:11.929483 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca" pod "bfb5e3be-84fc-41e3-adec-0848199a73cb" (UID: "bfb5e3be-84fc-41e3-adec-0848199a73cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:12.029175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.029138 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:12.029175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.029176 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb5e3be-84fc-41e3-adec-0848199a73cb-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:12.029354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.029187 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtln\" (UniqueName: \"kubernetes.io/projected/bfb5e3be-84fc-41e3-adec-0848199a73cb-kube-api-access-gxtln\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:12.692027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.691997 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" Nov 03 03:24:12.692200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.691993 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46" event={"ID":"bfb5e3be-84fc-41e3-adec-0848199a73cb","Type":"ContainerDied","Data":"a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0"} Nov 03 03:24:12.692200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:12.692143 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b83bf7cc06c2ac99dd75e86356ffab3c0c8902834be2ec11eb13368358f4f0" Nov 03 03:24:14.826568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.826526 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74"] Nov 03 03:24:14.827560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.827533 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bfb5e3be-84fc-41e3-adec-0848199a73cb" containerName="revoke-keys" Nov 03 03:24:14.827676 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.827561 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb5e3be-84fc-41e3-adec-0848199a73cb" containerName="revoke-keys" Nov 03 03:24:14.827742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.827725 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb5e3be-84fc-41e3-adec-0848199a73cb" containerName="revoke-keys" Nov 03 03:24:14.831656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.831634 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:14.905240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.904826 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sr2x\" (UniqueName: \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:14.905240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.904941 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:14.905240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.904991 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:14.929814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.929784 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74"] Nov 03 03:24:14.936230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.936199 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:14.937013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.936985 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:14.937221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.937206 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:14.937420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:14.937406 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:15.007009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.006975 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sr2x\" (UniqueName: \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.008428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.008388 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.009556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.009536 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.010272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.009447 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.016004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.015957 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.071920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.071887 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sr2x\" (UniqueName: \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\") pod \"maas-api-revoke-sub-29efb7f0-kvw74\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.145630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.145538 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:15.531879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.531829 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74"] Nov 03 03:24:15.534122 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:15.534096 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a576e23_89de_440c_8335_1bbdaf924d23.slice/crio-973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3 WatchSource:0}: Error finding container 973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3: Status 404 returned error can't find the container with id 973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3 Nov 03 03:24:15.729105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:15.729071 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" event={"ID":"7a576e23-89de-440c-8335-1bbdaf924d23","Type":"ContainerStarted","Data":"973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3"} Nov 03 03:24:16.747274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:16.747238 5275 generic.go:356] "Generic (PLEG): container finished" podID="7a576e23-89de-440c-8335-1bbdaf924d23" containerID="a8381f6f6026cfea278d07ceb56c98056057a535213e0ceda9cca7da734aa841" exitCode=0 Nov 03 03:24:16.747691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:16.747307 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" event={"ID":"7a576e23-89de-440c-8335-1bbdaf924d23","Type":"ContainerDied","Data":"a8381f6f6026cfea278d07ceb56c98056057a535213e0ceda9cca7da734aa841"} Nov 03 03:24:17.924390 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:17.924358 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:18.056974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.056937 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\" (UniqueName: \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\") pod \"7a576e23-89de-440c-8335-1bbdaf924d23\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " Nov 03 03:24:18.057209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.057041 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\") pod \"7a576e23-89de-440c-8335-1bbdaf924d23\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " Nov 03 03:24:18.057285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.057212 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\") pod \"7a576e23-89de-440c-8335-1bbdaf924d23\" (UID: \"7a576e23-89de-440c-8335-1bbdaf924d23\") " Nov 03 03:24:18.058260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.058203 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca" pod "7a576e23-89de-440c-8335-1bbdaf924d23" (UID: "7a576e23-89de-440c-8335-1bbdaf924d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:18.075662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.075629 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca" pod "7a576e23-89de-440c-8335-1bbdaf924d23" (UID: "7a576e23-89de-440c-8335-1bbdaf924d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:18.092532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.092477 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x" pod "7a576e23-89de-440c-8335-1bbdaf924d23" (UID: "7a576e23-89de-440c-8335-1bbdaf924d23"). InnerVolumeSpecName "kube-api-access-4sr2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:18.159192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.159159 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:18.159192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.159188 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a576e23-89de-440c-8335-1bbdaf924d23-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:18.159192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.159199 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sr2x\" (UniqueName: \"kubernetes.io/projected/7a576e23-89de-440c-8335-1bbdaf924d23-kube-api-access-4sr2x\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:18.766301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.766272 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" Nov 03 03:24:18.766301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.766286 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74" event={"ID":"7a576e23-89de-440c-8335-1bbdaf924d23","Type":"ContainerDied","Data":"973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3"} Nov 03 03:24:18.766565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:18.766342 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="973e944a4f3f9bf5807b09326df9fea5dda06246fc1a1d9e2596a0eceb9904b3" Nov 03 03:24:19.051281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:19.051242 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97"] Nov 03 03:24:19.054365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:19.054342 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21587eec-w7z97"] Nov 03 03:24:20.984339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:20.984301 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97010889-70a4-4aef-8810-8ee36c1bda6d" path="/var/lib/kubelet/pods/97010889-70a4-4aef-8810-8ee36c1bda6d/volumes" Nov 03 03:24:24.442408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.442337 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v"] Nov 03 03:24:24.442910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.442870 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7a576e23-89de-440c-8335-1bbdaf924d23" containerName="revoke-keys" Nov 03 03:24:24.442910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.442883 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a576e23-89de-440c-8335-1bbdaf924d23" containerName="revoke-keys" Nov 03 03:24:24.443002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.442991 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a576e23-89de-440c-8335-1bbdaf924d23" containerName="revoke-keys" Nov 03 03:24:24.447600 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.447581 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.547326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.547292 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.547326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.547330 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcx4\" (UniqueName: \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.547555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.547529 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.549252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.549223 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:24.549252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.549243 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:24.549448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.549326 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:24.549448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.549363 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:24.552827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.552804 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v"] Nov 03 03:24:24.649222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.649188 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.649404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.649236 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcx4\" (UniqueName: \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.649404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.649372 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.650035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.650010 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.651751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.651733 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.711685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.711615 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcx4\" (UniqueName: \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\") pod \"maas-api-revoke-sub-34ce11a0-xqd5v\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:24.761144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:24.761108 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:25.142193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:25.142164 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v"] Nov 03 03:24:25.142632 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:25.142603 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60faca7e_f074_4b92_81d9_e6539c26f75c.slice/crio-f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300 WatchSource:0}: Error finding container f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300: Status 404 returned error can't find the container with id f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300 Nov 03 03:24:25.818191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:25.818149 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" event={"ID":"60faca7e-f074-4b92-81d9-e6539c26f75c","Type":"ContainerStarted","Data":"b06468cf1469de7aded9755825f40a2e6f706ea9cd208ad004d6b31b2a490ae7"} Nov 03 03:24:25.818709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:25.818212 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" event={"ID":"60faca7e-f074-4b92-81d9-e6539c26f75c","Type":"ContainerStarted","Data":"f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300"} Nov 03 03:24:25.937286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:25.937226 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" podStartSLOduration=1.937209948 podStartE2EDuration="1.937209948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:25.933390912 +0000 UTC m=+1357.086079334" watchObservedRunningTime="2026-11-03 03:24:25.937209948 +0000 UTC m=+1357.089898369" Nov 03 03:24:26.825994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:26.825961 5275 generic.go:356] "Generic (PLEG): container finished" podID="60faca7e-f074-4b92-81d9-e6539c26f75c" containerID="b06468cf1469de7aded9755825f40a2e6f706ea9cd208ad004d6b31b2a490ae7" exitCode=0 Nov 03 03:24:26.826417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:26.826051 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" event={"ID":"60faca7e-f074-4b92-81d9-e6539c26f75c","Type":"ContainerDied","Data":"b06468cf1469de7aded9755825f40a2e6f706ea9cd208ad004d6b31b2a490ae7"} Nov 03 03:24:27.999789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:27.999764 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:28.092180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.092073 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\") pod \"60faca7e-f074-4b92-81d9-e6539c26f75c\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " Nov 03 03:24:28.092180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.092149 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\" (UniqueName: \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\") pod \"60faca7e-f074-4b92-81d9-e6539c26f75c\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " Nov 03 03:24:28.092414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.092186 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\") pod \"60faca7e-f074-4b92-81d9-e6539c26f75c\" (UID: \"60faca7e-f074-4b92-81d9-e6539c26f75c\") " Nov 03 03:24:28.092502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.092465 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca" pod "60faca7e-f074-4b92-81d9-e6539c26f75c" (UID: "60faca7e-f074-4b92-81d9-e6539c26f75c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:28.092787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.092765 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:28.094499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.094469 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca" pod "60faca7e-f074-4b92-81d9-e6539c26f75c" (UID: "60faca7e-f074-4b92-81d9-e6539c26f75c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:28.094632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.094611 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4" pod "60faca7e-f074-4b92-81d9-e6539c26f75c" (UID: "60faca7e-f074-4b92-81d9-e6539c26f75c"). InnerVolumeSpecName "kube-api-access-cjcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:28.194387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.194358 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjcx4\" (UniqueName: \"kubernetes.io/projected/60faca7e-f074-4b92-81d9-e6539c26f75c-kube-api-access-cjcx4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:28.194387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.194386 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60faca7e-f074-4b92-81d9-e6539c26f75c-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:28.842206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.842177 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" Nov 03 03:24:28.842206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.842191 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v" event={"ID":"60faca7e-f074-4b92-81d9-e6539c26f75c","Type":"ContainerDied","Data":"f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300"} Nov 03 03:24:28.842421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:28.842238 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24589a96b01e0d521e1cb58fa20bb8f21a556fd9d793d0162ba8cfd76264300" Nov 03 03:24:31.233581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.233548 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745"] Nov 03 03:24:31.234140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.234125 5275 cpu_manager.go:414] "containerMap: removing container" podUID="60faca7e-f074-4b92-81d9-e6539c26f75c" containerName="revoke-keys" Nov 03 03:24:31.234185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.234142 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60faca7e-f074-4b92-81d9-e6539c26f75c" containerName="revoke-keys" Nov 03 03:24:31.234229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.234220 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="60faca7e-f074-4b92-81d9-e6539c26f75c" containerName="revoke-keys" Nov 03 03:24:31.237622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.237605 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.331335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.331305 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.331510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.331401 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqvk\" (UniqueName: \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.331575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.331562 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.340999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.340973 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:31.341164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.341031 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:31.341164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.341038 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:31.341164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.341051 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:31.344492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.344475 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745"] Nov 03 03:24:31.432811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.432771 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.433022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.432895 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqvk\" (UniqueName: \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.433106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.433073 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.433587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.433562 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.435639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.435618 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.494339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.494258 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqvk\" (UniqueName: \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\") pod \"maas-api-revoke-sub-48d36f1a-mr745\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.557944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.557908 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:31.940217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:31.940188 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745"] Nov 03 03:24:31.941036 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:31.941009 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378dd831_fa1d_480f_83aa_780d6e619eae.slice/crio-63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960 WatchSource:0}: Error finding container 63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960: Status 404 returned error can't find the container with id 63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960 Nov 03 03:24:32.332515 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:24:32.332482 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:24:32.880730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:32.880691 5275 generic.go:356] "Generic (PLEG): container finished" podID="378dd831-fa1d-480f-83aa-780d6e619eae" containerID="2a9089fc209ee0e972cea63233732369e8fbe275defb9c70a48bd716400586a6" exitCode=0 Nov 03 03:24:32.880963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:32.880732 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" event={"ID":"378dd831-fa1d-480f-83aa-780d6e619eae","Type":"ContainerDied","Data":"2a9089fc209ee0e972cea63233732369e8fbe275defb9c70a48bd716400586a6"} Nov 03 03:24:32.880963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:32.880783 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" event={"ID":"378dd831-fa1d-480f-83aa-780d6e619eae","Type":"ContainerStarted","Data":"63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960"} Nov 03 03:24:34.051490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.051460 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:34.166496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.166413 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\") pod \"378dd831-fa1d-480f-83aa-780d6e619eae\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " Nov 03 03:24:34.166801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.166547 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\") pod \"378dd831-fa1d-480f-83aa-780d6e619eae\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " Nov 03 03:24:34.166801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.166667 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\" (UniqueName: \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\") pod \"378dd831-fa1d-480f-83aa-780d6e619eae\" (UID: \"378dd831-fa1d-480f-83aa-780d6e619eae\") " Nov 03 03:24:34.168823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.168791 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca" pod "378dd831-fa1d-480f-83aa-780d6e619eae" (UID: "378dd831-fa1d-480f-83aa-780d6e619eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:34.171467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.171402 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca" pod "378dd831-fa1d-480f-83aa-780d6e619eae" (UID: "378dd831-fa1d-480f-83aa-780d6e619eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:34.171467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.171423 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk" pod "378dd831-fa1d-480f-83aa-780d6e619eae" (UID: "378dd831-fa1d-480f-83aa-780d6e619eae"). InnerVolumeSpecName "kube-api-access-tvqvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:34.172660 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.172637 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqvk\" (UniqueName: \"kubernetes.io/projected/378dd831-fa1d-480f-83aa-780d6e619eae-kube-api-access-tvqvk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:34.172768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.172664 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:34.172768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.172681 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378dd831-fa1d-480f-83aa-780d6e619eae-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:34.907545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.898371 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" Nov 03 03:24:34.907545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.898384 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745" event={"ID":"378dd831-fa1d-480f-83aa-780d6e619eae","Type":"ContainerDied","Data":"63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960"} Nov 03 03:24:34.907545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:34.898445 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ff306201262d21506d194c8f236a7b9a3bbf1705770445b228eaf1c086d960" Nov 03 03:24:35.491734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.491694 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:24:35.492605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.492581 5275 cpu_manager.go:414] "containerMap: removing container" podUID="378dd831-fa1d-480f-83aa-780d6e619eae" containerName="revoke-keys" Nov 03 03:24:35.492605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.492608 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378dd831-fa1d-480f-83aa-780d6e619eae" containerName="revoke-keys" Nov 03 03:24:35.492819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.492799 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="378dd831-fa1d-480f-83aa-780d6e619eae" containerName="revoke-keys" Nov 03 03:24:35.497243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.496978 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.597052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.596998 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.597387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.597361 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.597571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.597552 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.597735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.597716 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.597974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.597946 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkt6k\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.598132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.598115 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.598318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.598302 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.598474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.598458 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.598627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.598611 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.700960 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701044 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701142 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701190 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701225 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701256 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701339 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkt6k\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701379 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.701875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.701442 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.724304 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.724584 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.724908 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.725536 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.726468 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.726709 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.740487 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.774947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.748498 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:24:35.843872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.843305 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkt6k\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:35.847874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:35.844158 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-j457x\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:36.122213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.122110 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:36.727142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.727111 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:24:36.727841 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:36.727804 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7dd91e_595f_4629_86e0_803084960b13.slice/crio-ce8c1802e746474a682b55783f7fe8db5b9ae0a57525311ee692206a143fd417 WatchSource:0}: Error finding container ce8c1802e746474a682b55783f7fe8db5b9ae0a57525311ee692206a143fd417: Status 404 returned error can't find the container with id ce8c1802e746474a682b55783f7fe8db5b9ae0a57525311ee692206a143fd417 Nov 03 03:24:36.730465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.730427 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:36.730562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.730515 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:36.730622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.730563 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:36.951718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.951678 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" event={"ID":"3e7dd91e-595f-4629-86e0-803084960b13","Type":"ContainerStarted","Data":"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a"} Nov 03 03:24:36.951969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:36.951764 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" event={"ID":"3e7dd91e-595f-4629-86e0-803084960b13","Type":"ContainerStarted","Data":"ce8c1802e746474a682b55783f7fe8db5b9ae0a57525311ee692206a143fd417"} Nov 03 03:24:37.074119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:37.074067 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podStartSLOduration=2.074052389 podStartE2EDuration="2.074052389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:37.071341354 +0000 UTC m=+1368.224029774" watchObservedRunningTime="2026-11-03 03:24:37.074052389 +0000 UTC m=+1368.226740810" Nov 03 03:24:37.122943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:37.122905 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:37.124720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:37.124675 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:37.124956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:37.124766 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:38.124008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:38.123972 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:38.124420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:38.124026 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:38.944940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:38.944906 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 03:24:38.952874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:38.948954 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.065901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.057174 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"distinct-w4-b1ad4e-kserve-self-signed-certs\"" Nov 03 03:24:39.099082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.099034 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 03:24:39.119800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.119767 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.119961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.119816 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zjp\" (UniqueName: \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.120018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.119975 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.120228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.120206 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.120402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.120382 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.120448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.120424 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.123584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.123565 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:39.123673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.123617 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224200 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224289 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224317 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224396 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224432 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zjp\" (UniqueName: \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.224742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.224503 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.229874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.225733 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.229874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.225880 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.229874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.225951 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.233763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.233730 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.233953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.233769 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.292953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.292917 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zjp\" (UniqueName: \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\") pod \"distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.322389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.322351 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:39.808649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:39.808622 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 03:24:39.809112 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:39.809082 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e43148_cf6e_49cc_9c1c_1920b54edd8c.slice/crio-67fcf91ab9453d0a48a39fcb4f747bc0ca464aaf25d3368b2a4dc831d72939b7 WatchSource:0}: Error finding container 67fcf91ab9453d0a48a39fcb4f747bc0ca464aaf25d3368b2a4dc831d72939b7: Status 404 returned error can't find the container with id 67fcf91ab9453d0a48a39fcb4f747bc0ca464aaf25d3368b2a4dc831d72939b7 Nov 03 03:24:40.036330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.036288 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" event={"ID":"35e43148-cf6e-49cc-9c1c-1920b54edd8c","Type":"ContainerStarted","Data":"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab"} Nov 03 03:24:40.036531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.036341 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" event={"ID":"35e43148-cf6e-49cc-9c1c-1920b54edd8c","Type":"ContainerStarted","Data":"67fcf91ab9453d0a48a39fcb4f747bc0ca464aaf25d3368b2a4dc831d72939b7"} Nov 03 03:24:40.126234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.126142 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:40.126234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.126213 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:40.175873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.157639 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podStartSLOduration=2.157619308 podStartE2EDuration="2.157619308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:40.155238217 +0000 UTC m=+1371.307926643" watchObservedRunningTime="2026-11-03 03:24:40.157619308 +0000 UTC m=+1371.310307729" Nov 03 03:24:40.322569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.322509 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:40.899261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.899217 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7"] Nov 03 03:24:40.904701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.904678 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:40.998092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:40.998057 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7"] Nov 03 03:24:41.011039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.011007 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:41.011039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.011007 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:41.011378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.011008 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:41.011378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.011149 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:41.046662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.046627 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.046914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.046701 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.046914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.046766 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6kd\" (UniqueName: \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.123250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.123211 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:41.123424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.123275 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:41.148423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.148380 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6kd\" (UniqueName: \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.148618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.148580 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.148715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.148700 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.154929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.149357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.173874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.173060 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.209762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.209712 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6kd\" (UniqueName: \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\") pod \"maas-api-revoke-sub-17506388-v5tw7\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.219506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.219468 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:41.636249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.636221 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7"] Nov 03 03:24:41.636650 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:41.636627 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7143caa_f22e_4e54_ba7e_0e5199a4d940.slice/crio-ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060 WatchSource:0}: Error finding container ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060: Status 404 returned error can't find the container with id ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060 Nov 03 03:24:41.877408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.877367 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:24:41.884120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.884052 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.988708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.988667 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.988898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.988731 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.988898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.988763 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.988898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.988835 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.989121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.988937 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.989121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.989008 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.989121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.989055 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.989121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.989111 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.989345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.989147 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:41.997741 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:41.997710 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:24:42.077102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.077067 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" event={"ID":"a7143caa-f22e-4e54-ba7e-0e5199a4d940","Type":"ContainerStarted","Data":"ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060"} Nov 03 03:24:42.090329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090297 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090353 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090386 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090409 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090542 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090638 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090715 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.090958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.090773 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.091175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.091149 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.091270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.091197 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.091310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.091263 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.091780 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.091752 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.093266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.093242 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.093417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.093394 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.123379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.123351 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:42.123527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.123403 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:42.174901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.174695 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.176500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.176470 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.216385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.216353 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:42.678955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.678418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:24:42.680789 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:42.680760 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80512c30_b976_4e03_93d3_f106775a5d85.slice/crio-edc69b6c18bc38739a3cb774f45958803394a304093a362cc6e662236c6f21b3 WatchSource:0}: Error finding container edc69b6c18bc38739a3cb774f45958803394a304093a362cc6e662236c6f21b3: Status 404 returned error can't find the container with id edc69b6c18bc38739a3cb774f45958803394a304093a362cc6e662236c6f21b3 Nov 03 03:24:42.684003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.683964 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:42.684087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.684052 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:42.684123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:42.684098 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:24:43.087125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.087086 5275 generic.go:356] "Generic (PLEG): container finished" podID="a7143caa-f22e-4e54-ba7e-0e5199a4d940" containerID="987353185d775e7f47dfafb5c16038540ba8ff3277b23a4509d1a6ade87d6161" exitCode=0 Nov 03 03:24:43.087346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.087126 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" event={"ID":"a7143caa-f22e-4e54-ba7e-0e5199a4d940","Type":"ContainerDied","Data":"987353185d775e7f47dfafb5c16038540ba8ff3277b23a4509d1a6ade87d6161"} Nov 03 03:24:43.089312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.089283 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" event={"ID":"80512c30-b976-4e03-93d3-f106775a5d85","Type":"ContainerStarted","Data":"3991e9dc521e164e8929d45018879740ac98672d517335b9010e1b7e47b8e93c"} Nov 03 03:24:43.089437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.089331 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" event={"ID":"80512c30-b976-4e03-93d3-f106775a5d85","Type":"ContainerStarted","Data":"edc69b6c18bc38739a3cb774f45958803394a304093a362cc6e662236c6f21b3"} Nov 03 03:24:43.123129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.123085 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 03:24:43.123302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.123157 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 03:24:43.217107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.217069 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:43.218517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.218487 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:43.218659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.218593 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:43.328510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:43.328454 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podStartSLOduration=2.328439614 podStartE2EDuration="2.328439614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:43.325035332 +0000 UTC m=+1374.477723753" watchObservedRunningTime="2026-11-03 03:24:43.328439614 +0000 UTC m=+1374.481128035" Nov 03 03:24:44.219339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.218985 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:44.219339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.219051 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:44.400034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.399997 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:44.545888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.545125 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\") pod \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " Nov 03 03:24:44.545888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.545273 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\") pod \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " Nov 03 03:24:44.545888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.545364 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\" (UniqueName: \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\") pod \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\" (UID: \"a7143caa-f22e-4e54-ba7e-0e5199a4d940\") " Nov 03 03:24:44.546336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.546306 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca" pod "a7143caa-f22e-4e54-ba7e-0e5199a4d940" (UID: "a7143caa-f22e-4e54-ba7e-0e5199a4d940"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:44.554615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.553384 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd" pod "a7143caa-f22e-4e54-ba7e-0e5199a4d940" (UID: "a7143caa-f22e-4e54-ba7e-0e5199a4d940"). InnerVolumeSpecName "kube-api-access-ch6kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:44.554615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.553939 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca" pod "a7143caa-f22e-4e54-ba7e-0e5199a4d940" (UID: "a7143caa-f22e-4e54-ba7e-0e5199a4d940"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:44.611869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.611294 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:44.615255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.612621 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:44.616932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.616911 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:24:44.647495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.647241 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:44.647495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.647278 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7143caa-f22e-4e54-ba7e-0e5199a4d940-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:44.647495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:44.647294 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6kd\" (UniqueName: \"kubernetes.io/projected/a7143caa-f22e-4e54-ba7e-0e5199a4d940-kube-api-access-ch6kd\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:45.124281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:45.124235 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" event={"ID":"a7143caa-f22e-4e54-ba7e-0e5199a4d940","Type":"ContainerDied","Data":"ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060"} Nov 03 03:24:45.124431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:45.124287 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7" Nov 03 03:24:45.124431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:45.124290 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2f66fa7a9e676b6beb086233ee60f8bc25593cef34b7415785931dab4d5060" Nov 03 03:24:45.216959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:45.216926 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:45.217143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:45.216980 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:46.217226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:46.217192 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:46.217611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:46.217250 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:47.217265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:47.217228 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:47.217743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:47.217280 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:48.217736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.217704 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:48.218150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.217760 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:48.479954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.479833 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt"] Nov 03 03:24:48.480444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.480426 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a7143caa-f22e-4e54-ba7e-0e5199a4d940" containerName="revoke-keys" Nov 03 03:24:48.480444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.480443 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7143caa-f22e-4e54-ba7e-0e5199a4d940" containerName="revoke-keys" Nov 03 03:24:48.480628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.480582 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7143caa-f22e-4e54-ba7e-0e5199a4d940" containerName="revoke-keys" Nov 03 03:24:48.484401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.484378 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.518953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.518922 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.519185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.519162 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhghr\" (UniqueName: \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.519255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.519203 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.538697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.538662 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt"] Nov 03 03:24:48.585875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.585802 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:48.586067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.585883 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:48.586067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.585929 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:48.586671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.586645 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:48.620130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.620094 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.620292 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.620254 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhghr\" (UniqueName: \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.620292 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.620278 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.620829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.620811 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.622756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.622727 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.681964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.681934 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhghr\" (UniqueName: \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\") pod \"maas-api-revoke-sub-19617288-zf2rt\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:48.802954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:48.802914 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:49.192837 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:49.192804 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8842914_cf5e_4bee_8796_0d379b29a2d5.slice/crio-ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780 WatchSource:0}: Error finding container ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780: Status 404 returned error can't find the container with id ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780 Nov 03 03:24:49.194500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:49.194426 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt"] Nov 03 03:24:49.216862 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:49.216824 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:24:49.216975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:49.216889 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:24:49.323419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:49.323382 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:49.336369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:49.336343 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:50.054472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.054439 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8"] Nov 03 03:24:50.057804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.057775 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef89d962-vfsq8"] Nov 03 03:24:50.167261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.167223 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" event={"ID":"b8842914-cf5e-4bee-8796-0d379b29a2d5","Type":"ContainerStarted","Data":"34c58cd6ae37a857ca87c9fbc4bfd06d9e4e94a410d982dff53e1f968af6d426"} Nov 03 03:24:50.167470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.167289 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" event={"ID":"b8842914-cf5e-4bee-8796-0d379b29a2d5","Type":"ContainerStarted","Data":"ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780"} Nov 03 03:24:50.181376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.181347 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:24:50.225783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.225754 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:50.226161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.226135 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:50.226904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.226887 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:24:50.284756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.284692 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" podStartSLOduration=2.284678268 podStartE2EDuration="2.284678268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:50.283027746 +0000 UTC m=+1381.435716167" watchObservedRunningTime="2026-11-03 03:24:50.284678268 +0000 UTC m=+1381.437366689" Nov 03 03:24:50.987610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:50.987573 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82abd35a-0fc4-451f-9db7-a96923917f82" path="/var/lib/kubelet/pods/82abd35a-0fc4-451f-9db7-a96923917f82/volumes" Nov 03 03:24:51.180058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.180022 5275 generic.go:356] "Generic (PLEG): container finished" podID="b8842914-cf5e-4bee-8796-0d379b29a2d5" containerID="34c58cd6ae37a857ca87c9fbc4bfd06d9e4e94a410d982dff53e1f968af6d426" exitCode=0 Nov 03 03:24:51.180256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.180101 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" event={"ID":"b8842914-cf5e-4bee-8796-0d379b29a2d5","Type":"ContainerDied","Data":"34c58cd6ae37a857ca87c9fbc4bfd06d9e4e94a410d982dff53e1f968af6d426"} Nov 03 03:24:51.660263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.660222 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm"] Nov 03 03:24:51.665430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.665405 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.721112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.721085 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm"] Nov 03 03:24:51.769107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.769078 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk75\" (UniqueName: \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.769267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.769194 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.769343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.769331 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.870919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.870881 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.871126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.871060 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.871194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.871160 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk75\" (UniqueName: \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.871729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.871703 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.873738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.873718 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.932649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.932566 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk75\" (UniqueName: \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\") pod \"maas-api-revoke-sub-ff1bad90-sflsm\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:51.980197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:51.980160 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:52.345231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.345203 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:52.362692 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:24:52.362657 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e0fac2_3a39_43a7_8fc1_d57bf3060852.slice/crio-d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7 WatchSource:0}: Error finding container d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7: Status 404 returned error can't find the container with id d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7 Nov 03 03:24:52.363049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.363022 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm"] Nov 03 03:24:52.377924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.377890 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\") pod \"b8842914-cf5e-4bee-8796-0d379b29a2d5\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " Nov 03 03:24:52.378093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.378008 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\" (UniqueName: \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\") pod \"b8842914-cf5e-4bee-8796-0d379b29a2d5\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " Nov 03 03:24:52.378157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.378131 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\") pod \"b8842914-cf5e-4bee-8796-0d379b29a2d5\" (UID: \"b8842914-cf5e-4bee-8796-0d379b29a2d5\") " Nov 03 03:24:52.378587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.378556 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca" pod "b8842914-cf5e-4bee-8796-0d379b29a2d5" (UID: "b8842914-cf5e-4bee-8796-0d379b29a2d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:52.380521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.380492 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca" pod "b8842914-cf5e-4bee-8796-0d379b29a2d5" (UID: "b8842914-cf5e-4bee-8796-0d379b29a2d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:52.380684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.380668 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr" pod "b8842914-cf5e-4bee-8796-0d379b29a2d5" (UID: "b8842914-cf5e-4bee-8796-0d379b29a2d5"). InnerVolumeSpecName "kube-api-access-vhghr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:52.480554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.480459 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:52.480554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.480500 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhghr\" (UniqueName: \"kubernetes.io/projected/b8842914-cf5e-4bee-8796-0d379b29a2d5-kube-api-access-vhghr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:52.480554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:52.480515 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8842914-cf5e-4bee-8796-0d379b29a2d5-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:53.203446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.203345 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" event={"ID":"f6e0fac2-3a39-43a7-8fc1-d57bf3060852","Type":"ContainerStarted","Data":"de23698ad29d810c1b74090509559fcd11dc43dd8c994aba688be5d007dc915b"} Nov 03 03:24:53.203446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.203402 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" event={"ID":"f6e0fac2-3a39-43a7-8fc1-d57bf3060852","Type":"ContainerStarted","Data":"d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7"} Nov 03 03:24:53.204859 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.204820 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" event={"ID":"b8842914-cf5e-4bee-8796-0d379b29a2d5","Type":"ContainerDied","Data":"ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780"} Nov 03 03:24:53.204961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.204867 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt" Nov 03 03:24:53.204961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.204887 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab47d058e6b26651874c2ac0f1e262cb0a2bb778f258e1b2afd10c03c300c780" Nov 03 03:24:53.320081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:53.320025 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" podStartSLOduration=2.32000904 podStartE2EDuration="2.32000904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:53.318805512 +0000 UTC m=+1384.471493933" watchObservedRunningTime="2026-11-03 03:24:53.32000904 +0000 UTC m=+1384.472697460" Nov 03 03:24:54.212770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:54.212735 5275 generic.go:356] "Generic (PLEG): container finished" podID="f6e0fac2-3a39-43a7-8fc1-d57bf3060852" containerID="de23698ad29d810c1b74090509559fcd11dc43dd8c994aba688be5d007dc915b" exitCode=0 Nov 03 03:24:54.213214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:54.212821 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" event={"ID":"f6e0fac2-3a39-43a7-8fc1-d57bf3060852","Type":"ContainerDied","Data":"de23698ad29d810c1b74090509559fcd11dc43dd8c994aba688be5d007dc915b"} Nov 03 03:24:56.206134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.206095 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:56.299990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.299952 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" event={"ID":"f6e0fac2-3a39-43a7-8fc1-d57bf3060852","Type":"ContainerDied","Data":"d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7"} Nov 03 03:24:56.300343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.300320 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43ec08ee26fbb533045f1b894c1acb9d0f1d0d13718d42a1f4c8cd6d84d52a7" Nov 03 03:24:56.300592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.300577 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm" Nov 03 03:24:56.311762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.311726 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\") pod \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " Nov 03 03:24:56.312443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.312421 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\" (UniqueName: \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\") pod \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " Nov 03 03:24:56.313022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.312348 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca" pod "f6e0fac2-3a39-43a7-8fc1-d57bf3060852" (UID: "f6e0fac2-3a39-43a7-8fc1-d57bf3060852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:56.313239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.313221 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\") pod \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\" (UID: \"f6e0fac2-3a39-43a7-8fc1-d57bf3060852\") " Nov 03 03:24:56.319752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.319730 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:56.356083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.343328 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca" pod "f6e0fac2-3a39-43a7-8fc1-d57bf3060852" (UID: "f6e0fac2-3a39-43a7-8fc1-d57bf3060852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:56.385913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.384391 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75" pod "f6e0fac2-3a39-43a7-8fc1-d57bf3060852" (UID: "f6e0fac2-3a39-43a7-8fc1-d57bf3060852"). InnerVolumeSpecName "kube-api-access-wjk75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:56.487873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.481552 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:56.487873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.481604 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk75\" (UniqueName: \"kubernetes.io/projected/f6e0fac2-3a39-43a7-8fc1-d57bf3060852-kube-api-access-wjk75\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:24:56.643027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.642926 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:56.760031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.759981 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:56.962300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:56.962195 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:24:57.016029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.015968 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.101027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.100971 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.176885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.173256 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.266942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.264643 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.285887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.284948 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.645999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.645947 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:57.647098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.647063 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:24:57.829031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.829001 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:24:57.978989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.966286 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:24:57.978989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:57.978146 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:24:58.016891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.016674 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:24:58.025933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.023378 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:24:58.176874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.174020 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:24:58.176874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.174306 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:58.176874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.175366 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:24:58.270935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.267351 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:24:58.270935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.268032 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:24:58.409794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.383438 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:24:58.409794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.383501 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:24:58.422919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.410819 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:24:58.422919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:58.416007 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:24:59.318305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.318271 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7"] Nov 03 03:24:59.318842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.318822 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b8842914-cf5e-4bee-8796-0d379b29a2d5" containerName="revoke-keys" Nov 03 03:24:59.318842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.318838 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8842914-cf5e-4bee-8796-0d379b29a2d5" containerName="revoke-keys" Nov 03 03:24:59.319036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.318904 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f6e0fac2-3a39-43a7-8fc1-d57bf3060852" containerName="revoke-keys" Nov 03 03:24:59.319036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.318912 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e0fac2-3a39-43a7-8fc1-d57bf3060852" containerName="revoke-keys" Nov 03 03:24:59.319125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.319065 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8842914-cf5e-4bee-8796-0d379b29a2d5" containerName="revoke-keys" Nov 03 03:24:59.319125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.319083 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e0fac2-3a39-43a7-8fc1-d57bf3060852" containerName="revoke-keys" Nov 03 03:24:59.323195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.323173 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.375925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.375882 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:59.376265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.376191 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:59.376386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.376302 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:59.376386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.376344 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:24:59.380951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.380914 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7"] Nov 03 03:24:59.416051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.416012 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.416243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.416160 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.416353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.416327 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljn9\" (UniqueName: \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.518396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.518349 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.518588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.518527 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.518655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.518644 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljn9\" (UniqueName: \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.519167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.519128 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.521456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.521423 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.580890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.580771 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljn9\" (UniqueName: \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\") pod \"maas-api-revoke-sub-e008e559-4gxc7\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:24:59.640063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:24:59.640027 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:25:00.024933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:00.024903 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7"] Nov 03 03:25:00.026074 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:00.026048 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea8aca3_9a0a_49b3_ba03_d1c6d35bc4a2.slice/crio-33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189 WatchSource:0}: Error finding container 33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189: Status 404 returned error can't find the container with id 33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189 Nov 03 03:25:00.465124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:00.465087 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" event={"ID":"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2","Type":"ContainerStarted","Data":"298a4c1c8e5721d8bbbf586f8d87f9e33217594432a49faf0bf8ce7a89d9ca4f"} Nov 03 03:25:00.465582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:00.465152 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" event={"ID":"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2","Type":"ContainerStarted","Data":"33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189"} Nov 03 03:25:00.581405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:00.581346 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" podStartSLOduration=1.581330585 podStartE2EDuration="1.581330585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:00.580066689 +0000 UTC m=+1391.732755110" watchObservedRunningTime="2026-11-03 03:25:00.581330585 +0000 UTC m=+1391.734019006" Nov 03 03:25:01.476194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:01.476154 5275 generic.go:356] "Generic (PLEG): container finished" podID="5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" containerID="298a4c1c8e5721d8bbbf586f8d87f9e33217594432a49faf0bf8ce7a89d9ca4f" exitCode=0 Nov 03 03:25:01.476668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:01.476245 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" event={"ID":"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2","Type":"ContainerDied","Data":"298a4c1c8e5721d8bbbf586f8d87f9e33217594432a49faf0bf8ce7a89d9ca4f"} Nov 03 03:25:02.334477 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:25:02.334436 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:25:02.658207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.658174 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:25:02.761092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.761062 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\" (UniqueName: \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\") pod \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " Nov 03 03:25:02.761240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.761168 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\") pod \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " Nov 03 03:25:02.761301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.761286 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\") pod \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\" (UID: \"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2\") " Nov 03 03:25:02.761691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.761662 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca" pod "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" (UID: "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:02.761991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.761905 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:02.763306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.763276 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca" pod "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" (UID: "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:02.763517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.763500 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9" pod "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" (UID: "5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2"). InnerVolumeSpecName "kube-api-access-sljn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:02.862710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.862661 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljn9\" (UniqueName: \"kubernetes.io/projected/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-kube-api-access-sljn9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:02.862710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:02.862697 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:03.495405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:03.495370 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" Nov 03 03:25:03.495405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:03.495384 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7" event={"ID":"5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2","Type":"ContainerDied","Data":"33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189"} Nov 03 03:25:03.495628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:03.495462 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b066bd86dacaa89ae64d73cdf256a88b5dc15571b201d0069016da6c78d189" Nov 03 03:25:04.058190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:04.058133 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h"] Nov 03 03:25:04.061264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:04.061235 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3fc7d7-c994h"] Nov 03 03:25:04.986733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:04.986702 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3" path="/var/lib/kubelet/pods/f782a5ae-1f3c-4c4f-a10b-e51c5fe5d9c3/volumes" Nov 03 03:25:06.049765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:06.049729 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s"] Nov 03 03:25:06.099254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:06.099219 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877156c9-lk55s"] Nov 03 03:25:06.984743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:06.984709 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1" path="/var/lib/kubelet/pods/e2f7d56e-56b7-4b04-9cc5-de7d8a5325b1/volumes" Nov 03 03:25:09.959012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:09.958972 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh"] Nov 03 03:25:09.959520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:09.959506 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" containerName="revoke-keys" Nov 03 03:25:09.959520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:09.959520 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" containerName="revoke-keys" Nov 03 03:25:09.959673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:09.959662 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" containerName="revoke-keys" Nov 03 03:25:09.964536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:09.964497 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.021748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.021712 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh"] Nov 03 03:25:10.056505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.056466 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.056700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.056625 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.056700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.056658 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwq6\" (UniqueName: \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.065226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.065199 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:25:10.065395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.065241 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:25:10.065395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.065272 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:25:10.065684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.065668 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:25:10.158144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.158108 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwq6\" (UniqueName: \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.158333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.158320 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.158452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.158437 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.159161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.159105 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.160875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.160830 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.219645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.219572 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwq6\" (UniqueName: \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\") pod \"maas-api-revoke-sub-aaa7e47a-q2jjh\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.280913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.280870 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:10.666179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:10.666140 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh"] Nov 03 03:25:10.667747 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:10.667720 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88318b6_e3d0_4df0_98bd_eb4624471977.slice/crio-2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf WatchSource:0}: Error finding container 2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf: Status 404 returned error can't find the container with id 2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf Nov 03 03:25:11.560310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:11.560270 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" event={"ID":"c88318b6-e3d0-4df0-98bd-eb4624471977","Type":"ContainerStarted","Data":"a7612005487f8fad36a6e58afba01ecb053f9426374a283df17a0cf87134a7df"} Nov 03 03:25:11.560919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:11.560358 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" event={"ID":"c88318b6-e3d0-4df0-98bd-eb4624471977","Type":"ContainerStarted","Data":"2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf"} Nov 03 03:25:11.677286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:11.677236 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" podStartSLOduration=2.677217282 podStartE2EDuration="2.677217282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:11.675466235 +0000 UTC m=+1402.828154656" watchObservedRunningTime="2026-11-03 03:25:11.677217282 +0000 UTC m=+1402.829905799" Nov 03 03:25:12.568879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:12.568825 5275 generic.go:356] "Generic (PLEG): container finished" podID="c88318b6-e3d0-4df0-98bd-eb4624471977" containerID="a7612005487f8fad36a6e58afba01ecb053f9426374a283df17a0cf87134a7df" exitCode=0 Nov 03 03:25:12.569346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:12.568918 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" event={"ID":"c88318b6-e3d0-4df0-98bd-eb4624471977","Type":"ContainerDied","Data":"a7612005487f8fad36a6e58afba01ecb053f9426374a283df17a0cf87134a7df"} Nov 03 03:25:14.326067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.326031 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:14.521767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.521725 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\") pod \"c88318b6-e3d0-4df0-98bd-eb4624471977\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " Nov 03 03:25:14.522494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.522471 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\" (UniqueName: \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\") pod \"c88318b6-e3d0-4df0-98bd-eb4624471977\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " Nov 03 03:25:14.527993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.527965 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\") pod \"c88318b6-e3d0-4df0-98bd-eb4624471977\" (UID: \"c88318b6-e3d0-4df0-98bd-eb4624471977\") " Nov 03 03:25:14.557917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.556478 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca" pod "c88318b6-e3d0-4df0-98bd-eb4624471977" (UID: "c88318b6-e3d0-4df0-98bd-eb4624471977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:14.557917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.557627 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca" pod "c88318b6-e3d0-4df0-98bd-eb4624471977" (UID: "c88318b6-e3d0-4df0-98bd-eb4624471977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:14.601932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.561340 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6" pod "c88318b6-e3d0-4df0-98bd-eb4624471977" (UID: "c88318b6-e3d0-4df0-98bd-eb4624471977"). InnerVolumeSpecName "kube-api-access-jpwq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:14.667872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.667635 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:14.667872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.667673 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwq6\" (UniqueName: \"kubernetes.io/projected/c88318b6-e3d0-4df0-98bd-eb4624471977-kube-api-access-jpwq6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:14.667872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.667691 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88318b6-e3d0-4df0-98bd-eb4624471977-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:14.671868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.670911 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" event={"ID":"c88318b6-e3d0-4df0-98bd-eb4624471977","Type":"ContainerDied","Data":"2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf"} Nov 03 03:25:14.671868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.671054 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2143ca26e6613bdd98492382f6e87eb7cf470a10b8523d68afa4c297794d90bf" Nov 03 03:25:14.671868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.671160 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh" Nov 03 03:25:14.832003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:14.831948 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:15.095889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:15.075870 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 03:25:17.910376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.910318 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2"] Nov 03 03:25:17.912151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.912121 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c88318b6-e3d0-4df0-98bd-eb4624471977" containerName="revoke-keys" Nov 03 03:25:17.912299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.912287 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88318b6-e3d0-4df0-98bd-eb4624471977" containerName="revoke-keys" Nov 03 03:25:17.912840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.912817 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88318b6-e3d0-4df0-98bd-eb4624471977" containerName="revoke-keys" Nov 03 03:25:17.917508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.917489 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:17.973289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:17.973244 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2"] Nov 03 03:25:18.014501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.014473 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:25:18.014501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.014484 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:25:18.014729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.014527 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:25:18.014729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.014565 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:25:18.073093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.073049 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.073308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.073114 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbh84\" (UniqueName: \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.073415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.073398 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.175491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.175395 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.175671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.175522 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.175671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.175568 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbh84\" (UniqueName: \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.176487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.176459 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.178204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.178178 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.237144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.237114 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbh84\" (UniqueName: \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\") pod \"maas-api-revoke-sub-f08c166b-7mlb2\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.531602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.531571 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:18.932473 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:18.932438 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a6b169_6fbb_4517_94fb_dd10405f103e.slice/crio-a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b WatchSource:0}: Error finding container a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b: Status 404 returned error can't find the container with id a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b Nov 03 03:25:18.935730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:18.935703 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2"] Nov 03 03:25:19.226564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.226483 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:25:19.231817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.231792 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.315240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.315210 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:25:19.397013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.396981 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397024 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397142 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397181 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnlt\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397207 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397254 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397333 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.397492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.397479 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498391 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498442 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498487 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498521 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnlt\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498556 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498587 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.498718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498676 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498772 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498793 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498866 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498896 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.498937 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.499170 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.499460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.499441 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.501275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.501251 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.501543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.501442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.559786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.559754 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.559994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.559931 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnlt\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:19.710712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.710677 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" event={"ID":"a5a6b169-6fbb-4517-94fb-dd10405f103e","Type":"ContainerStarted","Data":"2b9767fe00a77d179f12dd774e9cd44704a007e3738f0e02dcb82d9e0697d609"} Nov 03 03:25:19.710946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.710747 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" event={"ID":"a5a6b169-6fbb-4517-94fb-dd10405f103e","Type":"ContainerStarted","Data":"a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b"} Nov 03 03:25:19.828912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.828840 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" podStartSLOduration=2.828825399 podStartE2EDuration="2.828825399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:19.825106434 +0000 UTC m=+1410.977794856" watchObservedRunningTime="2026-11-03 03:25:19.828825399 +0000 UTC m=+1410.981513819" Nov 03 03:25:19.846867 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:19.846825 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:20.295585 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:20.295552 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6ceaeb_f9a3_40f7_b907_c039719e5986.slice/crio-6d4812bef66931a8968dce24137703a707e892e1a0370a936101883482b3085c WatchSource:0}: Error finding container 6d4812bef66931a8968dce24137703a707e892e1a0370a936101883482b3085c: Status 404 returned error can't find the container with id 6d4812bef66931a8968dce24137703a707e892e1a0370a936101883482b3085c Nov 03 03:25:20.296045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.295588 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:25:20.298278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.298237 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:20.298391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.298323 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:20.298466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.298369 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:20.721398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.721363 5275 generic.go:356] "Generic (PLEG): container finished" podID="a5a6b169-6fbb-4517-94fb-dd10405f103e" containerID="2b9767fe00a77d179f12dd774e9cd44704a007e3738f0e02dcb82d9e0697d609" exitCode=0 Nov 03 03:25:20.721603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.721477 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" event={"ID":"a5a6b169-6fbb-4517-94fb-dd10405f103e","Type":"ContainerDied","Data":"2b9767fe00a77d179f12dd774e9cd44704a007e3738f0e02dcb82d9e0697d609"} Nov 03 03:25:20.723678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.723583 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" event={"ID":"1d6ceaeb-f9a3-40f7-b907-c039719e5986","Type":"ContainerStarted","Data":"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c"} Nov 03 03:25:20.723678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.723635 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" event={"ID":"1d6ceaeb-f9a3-40f7-b907-c039719e5986","Type":"ContainerStarted","Data":"6d4812bef66931a8968dce24137703a707e892e1a0370a936101883482b3085c"} Nov 03 03:25:20.728423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.728398 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp"] Nov 03 03:25:20.732544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.732529 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.789965 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.789928 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp"] Nov 03 03:25:20.816882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.815523 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgtp\" (UniqueName: \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.816882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.815684 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.816882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.815780 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.860870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.847711 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:20.926003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.925959 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgtp\" (UniqueName: \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.926161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.926060 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.926161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.926102 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.926713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.926685 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.928657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.928630 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:20.976695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.976569 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podStartSLOduration=1.976547229 podStartE2EDuration="1.976547229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:20.960193217 +0000 UTC m=+1412.112881649" watchObservedRunningTime="2026-11-03 03:25:20.976547229 +0000 UTC m=+1412.129235652" Nov 03 03:25:20.987295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:20.987260 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgtp\" (UniqueName: \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\") pod \"maas-api-revoke-sub-3a7224be-q55zp\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:21.046810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.046773 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:21.483969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.483941 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp"] Nov 03 03:25:21.732046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.731961 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" event={"ID":"6c311e4e-2879-4c34-a671-225732c55c28","Type":"ContainerStarted","Data":"c7789d1c18bd3881f2001719adb5ef1a1890f630fb582ef0fe64f2f97bb29811"} Nov 03 03:25:21.848880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.848276 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:21.848880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.848333 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:21.940131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:21.940103 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:22.041112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.041082 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\") pod \"a5a6b169-6fbb-4517-94fb-dd10405f103e\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " Nov 03 03:25:22.041325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.041135 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\" (UniqueName: \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\") pod \"a5a6b169-6fbb-4517-94fb-dd10405f103e\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " Nov 03 03:25:22.041325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.041262 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\") pod \"a5a6b169-6fbb-4517-94fb-dd10405f103e\" (UID: \"a5a6b169-6fbb-4517-94fb-dd10405f103e\") " Nov 03 03:25:22.041588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.041556 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca" pod "a5a6b169-6fbb-4517-94fb-dd10405f103e" (UID: "a5a6b169-6fbb-4517-94fb-dd10405f103e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:22.041746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.041729 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:22.043379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.043361 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84" pod "a5a6b169-6fbb-4517-94fb-dd10405f103e" (UID: "a5a6b169-6fbb-4517-94fb-dd10405f103e"). InnerVolumeSpecName "kube-api-access-sbh84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:22.043453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.043372 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca" pod "a5a6b169-6fbb-4517-94fb-dd10405f103e" (UID: "a5a6b169-6fbb-4517-94fb-dd10405f103e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:22.142820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.142786 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a6b169-6fbb-4517-94fb-dd10405f103e-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:22.142820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.142813 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbh84\" (UniqueName: \"kubernetes.io/projected/a5a6b169-6fbb-4517-94fb-dd10405f103e-kube-api-access-sbh84\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:22.741552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.741515 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" Nov 03 03:25:22.742037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.741546 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2" event={"ID":"a5a6b169-6fbb-4517-94fb-dd10405f103e","Type":"ContainerDied","Data":"a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b"} Nov 03 03:25:22.742037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.741604 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a483c84af684982826f6ee9d5d66c157db5fc60301a5a5a5812da14b7727901b" Nov 03 03:25:22.743472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.743440 5275 generic.go:356] "Generic (PLEG): container finished" podID="6c311e4e-2879-4c34-a671-225732c55c28" containerID="793a3e44b95d02ff7ff514a4aa1ba21c5252d741e168a391e88c5e2e4cea38b8" exitCode=0 Nov 03 03:25:22.743613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.743500 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" event={"ID":"6c311e4e-2879-4c34-a671-225732c55c28","Type":"ContainerDied","Data":"793a3e44b95d02ff7ff514a4aa1ba21c5252d741e168a391e88c5e2e4cea38b8"} Nov 03 03:25:22.848180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.848148 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:22.848353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:22.848203 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:23.887316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:23.887279 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:23.895321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:23.895278 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:24.003872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.003037 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:24.080034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.079941 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz"] Nov 03 03:25:24.083482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.083459 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1b744a-cmxcz"] Nov 03 03:25:24.191195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.190549 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\") pod \"6c311e4e-2879-4c34-a671-225732c55c28\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " Nov 03 03:25:24.191195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.190648 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\" (UniqueName: \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\") pod \"6c311e4e-2879-4c34-a671-225732c55c28\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " Nov 03 03:25:24.191195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.190809 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\") pod \"6c311e4e-2879-4c34-a671-225732c55c28\" (UID: \"6c311e4e-2879-4c34-a671-225732c55c28\") " Nov 03 03:25:24.192538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.192231 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca" pod "6c311e4e-2879-4c34-a671-225732c55c28" (UID: "6c311e4e-2879-4c34-a671-225732c55c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:24.196762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.195830 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca" pod "6c311e4e-2879-4c34-a671-225732c55c28" (UID: "6c311e4e-2879-4c34-a671-225732c55c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:24.198394 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.198316 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp" pod "6c311e4e-2879-4c34-a671-225732c55c28" (UID: "6c311e4e-2879-4c34-a671-225732c55c28"). InnerVolumeSpecName "kube-api-access-qfgtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:24.293129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.293086 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:24.293129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.293121 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgtp\" (UniqueName: \"kubernetes.io/projected/6c311e4e-2879-4c34-a671-225732c55c28-kube-api-access-qfgtp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:24.293129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.293136 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c311e4e-2879-4c34-a671-225732c55c28-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:24.781413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.781381 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" Nov 03 03:25:24.781593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.781420 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp" event={"ID":"6c311e4e-2879-4c34-a671-225732c55c28","Type":"ContainerDied","Data":"c7789d1c18bd3881f2001719adb5ef1a1890f630fb582ef0fe64f2f97bb29811"} Nov 03 03:25:24.781593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.781501 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7789d1c18bd3881f2001719adb5ef1a1890f630fb582ef0fe64f2f97bb29811" Nov 03 03:25:24.848101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.848069 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:24.848317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.848123 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:24.985469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:24.985430 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af78eecf-a94b-41e3-80ce-fb8ee91ef4e9" path="/var/lib/kubelet/pods/af78eecf-a94b-41e3-80ce-fb8ee91ef4e9/volumes" Nov 03 03:25:25.847566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:25.847529 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:25.847760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:25.847599 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:26.847372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:26.847340 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 03:25:26.847890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:26.847407 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 03:25:27.234673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.234545 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:25:27.235213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235187 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a5a6b169-6fbb-4517-94fb-dd10405f103e" containerName="revoke-keys" Nov 03 03:25:27.235213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235206 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a6b169-6fbb-4517-94fb-dd10405f103e" containerName="revoke-keys" Nov 03 03:25:27.235455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235234 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6c311e4e-2879-4c34-a671-225732c55c28" containerName="revoke-keys" Nov 03 03:25:27.235455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235239 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c311e4e-2879-4c34-a671-225732c55c28" containerName="revoke-keys" Nov 03 03:25:27.235455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235315 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a6b169-6fbb-4517-94fb-dd10405f103e" containerName="revoke-keys" Nov 03 03:25:27.235455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.235327 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c311e4e-2879-4c34-a671-225732c55c28" containerName="revoke-keys" Nov 03 03:25:27.240272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.240242 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.331602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.331568 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.333836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.333795 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.333875 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57st\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.333912 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.333966 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.334055 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.334081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.334138 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.334400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.334183 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.352121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.352082 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:25:27.435254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435211 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435304 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435330 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435373 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435411 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435600 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435618 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435646 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57st\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.435715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435675 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.436024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.435672 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.436082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.436062 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.436566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.436540 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.436944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.436923 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.438055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.438026 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.439043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.439019 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.497205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.497123 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57st\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.498375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.498339 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.552948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.552915 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:25:27.555531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.555506 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:27.558192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.558172 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.622602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.622548 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:25:27.641840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.641783 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.641974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.641910 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.641974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.641941 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642008 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcgd\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642056 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642082 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642123 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642233 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.642328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.642311 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.743689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.743631 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcgd\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.743933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.743719 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.743933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.743748 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.743933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.743796 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.743932 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744005 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744098 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744161 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744193 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744226 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744287 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744518 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.744912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.744887 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.746709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.746676 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.746956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.746936 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.804789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.804756 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcgd\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.805006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.804984 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:27.851651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.851622 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:27.852085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.852064 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:27.852841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.852822 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:25:27.909839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:27.909804 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:28.013833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.013807 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:25:28.014550 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:28.014520 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d850884_8836_4708_8b5a_b6fe51ee418d.slice/crio-c4b7892a5c2b2f957517613e86d6fcd28b782d6d7acf225a064e68ad29c4dfb8 WatchSource:0}: Error finding container c4b7892a5c2b2f957517613e86d6fcd28b782d6d7acf225a064e68ad29c4dfb8: Status 404 returned error can't find the container with id c4b7892a5c2b2f957517613e86d6fcd28b782d6d7acf225a064e68ad29c4dfb8 Nov 03 03:25:28.018762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.018727 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.018897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.018817 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.018897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.018889 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.297893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.297861 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:25:28.318441 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:28.318399 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda071498d_79f4_40c0_b957_ed5a446f2858.slice/crio-3e231c67a0da5ae4925d272ae452b372085e57e02af21086a08d1c9378c627e9 WatchSource:0}: Error finding container 3e231c67a0da5ae4925d272ae452b372085e57e02af21086a08d1c9378c627e9: Status 404 returned error can't find the container with id 3e231c67a0da5ae4925d272ae452b372085e57e02af21086a08d1c9378c627e9 Nov 03 03:25:28.322476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.322436 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.322603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.322544 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.322603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.322577 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:28.632157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.632076 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc"] Nov 03 03:25:28.637559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.637534 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.662880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.658445 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.662880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.658571 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.662880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.658636 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvvc\" (UniqueName: \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.692873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.689974 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:25:28.692873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.690021 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:25:28.692873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.690074 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:25:28.692873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.690136 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:25:28.695904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.694941 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc"] Nov 03 03:25:28.793177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.791662 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.793177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.791790 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.793177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.791927 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvvc\" (UniqueName: \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.799923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.793809 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.804907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.803276 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.817686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.817637 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" event={"ID":"a071498d-79f4-40c0-b957-ed5a446f2858","Type":"ContainerStarted","Data":"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd"} Nov 03 03:25:28.817891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.817714 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" event={"ID":"a071498d-79f4-40c0-b957-ed5a446f2858","Type":"ContainerStarted","Data":"3e231c67a0da5ae4925d272ae452b372085e57e02af21086a08d1c9378c627e9"} Nov 03 03:25:28.820445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.820407 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" event={"ID":"7d850884-8836-4708-8b5a-b6fe51ee418d","Type":"ContainerStarted","Data":"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b"} Nov 03 03:25:28.820604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.820479 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" event={"ID":"7d850884-8836-4708-8b5a-b6fe51ee418d","Type":"ContainerStarted","Data":"c4b7892a5c2b2f957517613e86d6fcd28b782d6d7acf225a064e68ad29c4dfb8"} Nov 03 03:25:28.881886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.874054 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvvc\" (UniqueName: \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\") pod \"maas-api-revoke-sub-6b1bc38f-mp6zc\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:28.910489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.910342 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:28.947708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.947573 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podStartSLOduration=1.947550106 podStartE2EDuration="1.947550106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:28.939673915 +0000 UTC m=+1420.092362337" watchObservedRunningTime="2026-11-03 03:25:28.947550106 +0000 UTC m=+1420.100238530" Nov 03 03:25:28.965713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:28.964176 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:29.102951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.099661 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podStartSLOduration=2.09964262 podStartE2EDuration="2.09964262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:29.093372802 +0000 UTC m=+1420.246061216" watchObservedRunningTime="2026-11-03 03:25:29.09964262 +0000 UTC m=+1420.252331041" Nov 03 03:25:29.447017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.446989 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc"] Nov 03 03:25:29.589562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.582402 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:29.610078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.610041 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:29.610364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.610334 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:29.830586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.830552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" event={"ID":"474530e2-6abf-43ee-8be2-c0dd4ddec402","Type":"ContainerStarted","Data":"431a4dd33de9969db3656a0dd749016197ccd261c0779018f08f75dd542b2a73"} Nov 03 03:25:29.907901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.907482 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd"] Nov 03 03:25:29.912359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.912329 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:29.912483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.912374 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:29.914078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.914062 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:29.927237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.925016 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:29.927237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.925109 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:29.927237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.925192 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tsw\" (UniqueName: \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:29.944369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:29.944279 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd"] Nov 03 03:25:30.026590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.026556 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.026788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.026619 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.026788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.026690 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tsw\" (UniqueName: \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.027498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.027473 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.030368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.030340 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.118959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.114793 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tsw\" (UniqueName: \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\") pod \"maas-api-revoke-sub-2eb37775-dv9vd\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.235373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.235333 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:30.572874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.563670 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:30.572874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.563733 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:30.730478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.730439 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd"] Nov 03 03:25:30.730742 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:30.730704 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677722d4_2b90_4d36_b10c_8115a99c16e2.slice/crio-f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b WatchSource:0}: Error finding container f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b: Status 404 returned error can't find the container with id f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b Nov 03 03:25:30.844588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.844556 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" event={"ID":"677722d4-2b90-4d36-b10c-8115a99c16e2","Type":"ContainerStarted","Data":"f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b"} Nov 03 03:25:30.846483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.846454 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" event={"ID":"474530e2-6abf-43ee-8be2-c0dd4ddec402","Type":"ContainerStarted","Data":"28e394adfda1a0ef8b1a63247ad4ac6a8888b473e27c7fbddc79ee430d50a015"} Nov 03 03:25:30.911001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.910967 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:30.911403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.911016 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:30.975933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:30.968357 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" podStartSLOduration=2.96833804 podStartE2EDuration="2.96833804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:30.966321751 +0000 UTC m=+1422.119010174" watchObservedRunningTime="2026-11-03 03:25:30.96833804 +0000 UTC m=+1422.121026463" Nov 03 03:25:31.580874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.560308 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:31.580874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.560368 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:31.905881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.905755 5275 generic.go:356] "Generic (PLEG): container finished" podID="474530e2-6abf-43ee-8be2-c0dd4ddec402" containerID="28e394adfda1a0ef8b1a63247ad4ac6a8888b473e27c7fbddc79ee430d50a015" exitCode=0 Nov 03 03:25:31.906120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.906095 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" event={"ID":"474530e2-6abf-43ee-8be2-c0dd4ddec402","Type":"ContainerDied","Data":"28e394adfda1a0ef8b1a63247ad4ac6a8888b473e27c7fbddc79ee430d50a015"} Nov 03 03:25:31.909308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.909256 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" event={"ID":"677722d4-2b90-4d36-b10c-8115a99c16e2","Type":"ContainerStarted","Data":"fd94be92959801348124fcb50857607465271b254a38cf7e89e757a1fe28605d"} Nov 03 03:25:31.910993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.910969 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:31.911105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:31.911016 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:32.344558 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:25:32.344523 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:25:32.568876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.559247 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:32.568876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.559311 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:32.912870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.912259 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:32.912870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.912318 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:32.927234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.927196 5275 generic.go:356] "Generic (PLEG): container finished" podID="677722d4-2b90-4d36-b10c-8115a99c16e2" containerID="fd94be92959801348124fcb50857607465271b254a38cf7e89e757a1fe28605d" exitCode=0 Nov 03 03:25:32.927708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:32.927688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" event={"ID":"677722d4-2b90-4d36-b10c-8115a99c16e2","Type":"ContainerDied","Data":"fd94be92959801348124fcb50857607465271b254a38cf7e89e757a1fe28605d"} Nov 03 03:25:33.410107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.410068 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:33.538436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.536305 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\") pod \"474530e2-6abf-43ee-8be2-c0dd4ddec402\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " Nov 03 03:25:33.538436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.536371 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\" (UniqueName: \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\") pod \"474530e2-6abf-43ee-8be2-c0dd4ddec402\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " Nov 03 03:25:33.538436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.536528 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\") pod \"474530e2-6abf-43ee-8be2-c0dd4ddec402\" (UID: \"474530e2-6abf-43ee-8be2-c0dd4ddec402\") " Nov 03 03:25:33.538436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.537554 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca" pod "474530e2-6abf-43ee-8be2-c0dd4ddec402" (UID: "474530e2-6abf-43ee-8be2-c0dd4ddec402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:33.542344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.542311 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc" pod "474530e2-6abf-43ee-8be2-c0dd4ddec402" (UID: "474530e2-6abf-43ee-8be2-c0dd4ddec402"). InnerVolumeSpecName "kube-api-access-tnvvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:33.546989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.546953 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca" pod "474530e2-6abf-43ee-8be2-c0dd4ddec402" (UID: "474530e2-6abf-43ee-8be2-c0dd4ddec402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:33.559175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.558317 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:33.559175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.558378 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:33.639230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.639125 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:33.639230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.639170 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnvvc\" (UniqueName: \"kubernetes.io/projected/474530e2-6abf-43ee-8be2-c0dd4ddec402-kube-api-access-tnvvc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:33.639230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.639186 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474530e2-6abf-43ee-8be2-c0dd4ddec402-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:33.919023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.916456 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:33.919023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.916513 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:33.936955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.936637 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" Nov 03 03:25:33.936955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.936677 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc" event={"ID":"474530e2-6abf-43ee-8be2-c0dd4ddec402","Type":"ContainerDied","Data":"431a4dd33de9969db3656a0dd749016197ccd261c0779018f08f75dd542b2a73"} Nov 03 03:25:33.936955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:33.936757 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="431a4dd33de9969db3656a0dd749016197ccd261c0779018f08f75dd542b2a73" Nov 03 03:25:34.438925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.438898 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:34.565806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.564304 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:25:34.565806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.564372 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 03:25:34.600554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.600436 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\" (UniqueName: \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\") pod \"677722d4-2b90-4d36-b10c-8115a99c16e2\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " Nov 03 03:25:34.600803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.600587 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\") pod \"677722d4-2b90-4d36-b10c-8115a99c16e2\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " Nov 03 03:25:34.600803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.600723 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\") pod \"677722d4-2b90-4d36-b10c-8115a99c16e2\" (UID: \"677722d4-2b90-4d36-b10c-8115a99c16e2\") " Nov 03 03:25:34.601482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.601439 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca" pod "677722d4-2b90-4d36-b10c-8115a99c16e2" (UID: "677722d4-2b90-4d36-b10c-8115a99c16e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:34.604048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.604015 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw" pod "677722d4-2b90-4d36-b10c-8115a99c16e2" (UID: "677722d4-2b90-4d36-b10c-8115a99c16e2"). InnerVolumeSpecName "kube-api-access-z4tsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:34.605399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.605316 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca" pod "677722d4-2b90-4d36-b10c-8115a99c16e2" (UID: "677722d4-2b90-4d36-b10c-8115a99c16e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:34.702802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.702764 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:34.702802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.702796 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tsw\" (UniqueName: \"kubernetes.io/projected/677722d4-2b90-4d36-b10c-8115a99c16e2-kube-api-access-z4tsw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:34.702802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.702809 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677722d4-2b90-4d36-b10c-8115a99c16e2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:34.910267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.910172 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:25:34.910267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.910236 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 03:25:34.946595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.946563 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" Nov 03 03:25:34.947157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.946559 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd" event={"ID":"677722d4-2b90-4d36-b10c-8115a99c16e2","Type":"ContainerDied","Data":"f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b"} Nov 03 03:25:34.947157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:34.946683 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a41f38c641fbe6d64679b9cd75040bcb49b7edfe229835f7e529ce77c8f52b" Nov 03 03:25:35.073943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:35.073899 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr"] Nov 03 03:25:35.077591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:35.077538 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce39bde-2s6qr"] Nov 03 03:25:35.957514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:35.957471 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:35.958894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:35.958871 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:35.962987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:35.962961 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:25:36.033891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:36.033832 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:36.034689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:36.034550 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:36.035635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:36.035612 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:25:36.986213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:36.986183 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc" path="/var/lib/kubelet/pods/9f364e1d-a5b4-4ed3-aa08-f2f1d1c49adc/volumes" Nov 03 03:25:40.191871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.191521 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:40.428390 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.426382 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:40.428390 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.427170 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:40.845862 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.845820 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m"] Nov 03 03:25:40.846467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846453 5275 cpu_manager.go:414] "containerMap: removing container" podUID="474530e2-6abf-43ee-8be2-c0dd4ddec402" containerName="revoke-keys" Nov 03 03:25:40.846510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846469 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474530e2-6abf-43ee-8be2-c0dd4ddec402" containerName="revoke-keys" Nov 03 03:25:40.846510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846496 5275 cpu_manager.go:414] "containerMap: removing container" podUID="677722d4-2b90-4d36-b10c-8115a99c16e2" containerName="revoke-keys" Nov 03 03:25:40.846510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846501 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677722d4-2b90-4d36-b10c-8115a99c16e2" containerName="revoke-keys" Nov 03 03:25:40.846607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846602 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="677722d4-2b90-4d36-b10c-8115a99c16e2" containerName="revoke-keys" Nov 03 03:25:40.846638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.846615 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="474530e2-6abf-43ee-8be2-c0dd4ddec402" containerName="revoke-keys" Nov 03 03:25:40.853020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.852367 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:40.905475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.905440 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:25:40.934919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.934880 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:40.935107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.935008 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjwr\" (UniqueName: \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:40.935190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.935163 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:40.954695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.954656 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:25:40.954695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.954658 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:25:40.954969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.954666 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:25:40.960517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:40.960489 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m"] Nov 03 03:25:41.036580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.036540 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.036759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.036646 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjwr\" (UniqueName: \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.036759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.036731 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.037900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.037871 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.039917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.039894 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.098202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.098129 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjwr\" (UniqueName: \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\") pod \"maas-api-revoke-sub-375aa143-hkf9m\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.173061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.173023 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:41.554633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:41.554603 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m"] Nov 03 03:25:41.555524 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:41.555494 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee72008_e392_48b3_aed9_53de33ebf731.slice/crio-a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730 WatchSource:0}: Error finding container a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730: Status 404 returned error can't find the container with id a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730 Nov 03 03:25:42.144153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.144057 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" event={"ID":"4ee72008-e392-48b3-aed9-53de33ebf731","Type":"ContainerStarted","Data":"f50ef4faea0fa5e69b23370e20a1fa5683c22f8b428bf288c376d66b1f684c34"} Nov 03 03:25:42.144153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.144127 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" event={"ID":"4ee72008-e392-48b3-aed9-53de33ebf731","Type":"ContainerStarted","Data":"a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730"} Nov 03 03:25:42.277061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.276992 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" podStartSLOduration=2.276970525 podStartE2EDuration="2.276970525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:42.270812887 +0000 UTC m=+1433.423501312" watchObservedRunningTime="2026-11-03 03:25:42.276970525 +0000 UTC m=+1433.429658949" Nov 03 03:25:42.345710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.345675 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:25:42.350344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.350316 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.410941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.410834 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:25:42.456087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.456053 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czv8l\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.456420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.456387 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.456681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.456664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.456799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.456784 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.456969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.456951 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.457165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.457150 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.457308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.457293 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.457426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.457414 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.457581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.457568 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.558985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.558933 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559024 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czv8l\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559055 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559182 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559199 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559223 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559274 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559316 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559358 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.559503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559456 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.560140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559756 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.560140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.559981 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.560140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.560056 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.560245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.560148 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.562211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.562187 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.562328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.562308 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.624924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.624888 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.625093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.625073 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czv8l\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:42.667137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:42.667047 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:43.124378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.124341 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:25:43.153547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.153512 5275 generic.go:356] "Generic (PLEG): container finished" podID="4ee72008-e392-48b3-aed9-53de33ebf731" containerID="f50ef4faea0fa5e69b23370e20a1fa5683c22f8b428bf288c376d66b1f684c34" exitCode=0 Nov 03 03:25:43.153742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.153552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" event={"ID":"4ee72008-e392-48b3-aed9-53de33ebf731","Type":"ContainerDied","Data":"f50ef4faea0fa5e69b23370e20a1fa5683c22f8b428bf288c376d66b1f684c34"} Nov 03 03:25:43.167188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.167135 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:43.167255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.167205 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:43.167255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.167235 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:25:43.912101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.912052 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:25:43.917739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.917717 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:43.976712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:43.976681 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:25:44.007193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.004969 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.007193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.005114 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.007193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.005279 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46xs\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.007193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.005371 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.107327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.107289 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.107536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.107394 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.107536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.107446 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46xs\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.107536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.107511 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.108303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.108221 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.108303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.108295 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.110020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.109998 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.166163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.166072 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" event={"ID":"a07bf80a-6769-4df6-9e65-bca09a30152a","Type":"ContainerStarted","Data":"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a"} Nov 03 03:25:44.166163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.166136 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" event={"ID":"a07bf80a-6769-4df6-9e65-bca09a30152a","Type":"ContainerStarted","Data":"7ee6f8e2a1e778e31cb744de7e641e481d3df708bd8024db47f3efbff3b359f5"} Nov 03 03:25:44.168754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.168724 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46xs\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\") pod \"payload-processing-external-model-6575c6b868-qcjcn\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.233824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.233774 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:44.295504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.294266 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podStartSLOduration=2.294247556 podStartE2EDuration="2.294247556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:44.294029659 +0000 UTC m=+1435.446718077" watchObservedRunningTime="2026-11-03 03:25:44.294247556 +0000 UTC m=+1435.446935978" Nov 03 03:25:44.359313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.355652 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26"] Nov 03 03:25:44.366729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.366699 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.492186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.491483 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26"] Nov 03 03:25:44.513221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.513194 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:44.527385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.527348 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.527625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.527607 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29d2h\" (UniqueName: \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.527705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.527674 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.629613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.629578 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\" (UniqueName: \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\") pod \"4ee72008-e392-48b3-aed9-53de33ebf731\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " Nov 03 03:25:44.632784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.632745 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr" pod "4ee72008-e392-48b3-aed9-53de33ebf731" (UID: "4ee72008-e392-48b3-aed9-53de33ebf731"). InnerVolumeSpecName "kube-api-access-5kjwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:44.633232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.633207 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\") pod \"4ee72008-e392-48b3-aed9-53de33ebf731\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " Nov 03 03:25:44.633495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.633476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\") pod \"4ee72008-e392-48b3-aed9-53de33ebf731\" (UID: \"4ee72008-e392-48b3-aed9-53de33ebf731\") " Nov 03 03:25:44.633806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.633787 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d2h\" (UniqueName: \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.633934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.633889 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.633934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.633909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca" pod "4ee72008-e392-48b3-aed9-53de33ebf731" (UID: "4ee72008-e392-48b3-aed9-53de33ebf731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:44.634052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.634005 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.634254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.634236 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjwr\" (UniqueName: \"kubernetes.io/projected/4ee72008-e392-48b3-aed9-53de33ebf731-kube-api-access-5kjwr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:44.634322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.634260 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:44.635083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.635058 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.727246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.648743 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.727246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.668224 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:44.727246 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:44.672983 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aaeb03_26af_4d79_99f9_28656a7dc626.slice/crio-2c4d802b68759c17925960d110e0d7a09c4d8d07012bbe023b00b1a32546a40e WatchSource:0}: Error finding container 2c4d802b68759c17925960d110e0d7a09c4d8d07012bbe023b00b1a32546a40e: Status 404 returned error can't find the container with id 2c4d802b68759c17925960d110e0d7a09c4d8d07012bbe023b00b1a32546a40e Nov 03 03:25:44.727246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.715902 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:44.727246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.715962 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:44.794873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.793972 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:25:44.795066 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.795020 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d2h\" (UniqueName: \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\") pod \"maas-api-revoke-sub-c71f4a70-dqq26\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.795150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.795120 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca" pod "4ee72008-e392-48b3-aed9-53de33ebf731" (UID: "4ee72008-e392-48b3-aed9-53de33ebf731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:44.812366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.811292 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:44.870890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:44.857170 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee72008-e392-48b3-aed9-53de33ebf731-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:45.191838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.191784 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" event={"ID":"08aaeb03-26af-4d79-99f9-28656a7dc626","Type":"ContainerStarted","Data":"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277"} Nov 03 03:25:45.192303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.191883 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:25:45.192303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.191909 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" event={"ID":"08aaeb03-26af-4d79-99f9-28656a7dc626","Type":"ContainerStarted","Data":"2c4d802b68759c17925960d110e0d7a09c4d8d07012bbe023b00b1a32546a40e"} Nov 03 03:25:45.194278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.194253 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" Nov 03 03:25:45.194421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.194283 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m" event={"ID":"4ee72008-e392-48b3-aed9-53de33ebf731","Type":"ContainerDied","Data":"a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730"} Nov 03 03:25:45.194421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.194327 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52abc455d688170050c7312475a3be964df505e14addb030b7d9c35449e1730" Nov 03 03:25:45.313271 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:45.313237 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b449cf6_1294_4b9b_9974_7553d623590a.slice/crio-7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2 WatchSource:0}: Error finding container 7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2: Status 404 returned error can't find the container with id 7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2 Nov 03 03:25:45.314487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.314469 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26"] Nov 03 03:25:45.319181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.319125 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" podStartSLOduration=2.319106798 podStartE2EDuration="2.319106798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:45.308594419 +0000 UTC m=+1436.461282843" watchObservedRunningTime="2026-11-03 03:25:45.319106798 +0000 UTC m=+1436.471795224" Nov 03 03:25:45.671873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.669301 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:45.671873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:45.669366 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:46.244580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:46.244547 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" event={"ID":"0b449cf6-1294-4b9b-9974-7553d623590a","Type":"ContainerStarted","Data":"5cfa8482c2c294805e3050b34e7077eb3eb8e21e21c4edf2a40d9c178d6b303c"} Nov 03 03:25:46.244987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:46.244607 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" event={"ID":"0b449cf6-1294-4b9b-9974-7553d623590a","Type":"ContainerStarted","Data":"7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2"} Nov 03 03:25:46.378715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:46.378638 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" podStartSLOduration=2.378616467 podStartE2EDuration="2.378616467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:46.368107355 +0000 UTC m=+1437.520795780" watchObservedRunningTime="2026-11-03 03:25:46.378616467 +0000 UTC m=+1437.531304889" Nov 03 03:25:46.674916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:46.674837 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:46.675114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:46.674932 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:47.287254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:47.287216 5275 generic.go:356] "Generic (PLEG): container finished" podID="0b449cf6-1294-4b9b-9974-7553d623590a" containerID="5cfa8482c2c294805e3050b34e7077eb3eb8e21e21c4edf2a40d9c178d6b303c" exitCode=0 Nov 03 03:25:47.287632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:47.287305 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" event={"ID":"0b449cf6-1294-4b9b-9974-7553d623590a","Type":"ContainerDied","Data":"5cfa8482c2c294805e3050b34e7077eb3eb8e21e21c4edf2a40d9c178d6b303c"} Nov 03 03:25:47.671457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:47.671356 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:47.671752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:47.671720 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:48.513805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.513774 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:48.582545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.582450 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\" (UniqueName: \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\") pod \"0b449cf6-1294-4b9b-9974-7553d623590a\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " Nov 03 03:25:48.582730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.582575 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\") pod \"0b449cf6-1294-4b9b-9974-7553d623590a\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " Nov 03 03:25:48.582730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.582631 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\") pod \"0b449cf6-1294-4b9b-9974-7553d623590a\" (UID: \"0b449cf6-1294-4b9b-9974-7553d623590a\") " Nov 03 03:25:48.582989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.582959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca" pod "0b449cf6-1294-4b9b-9974-7553d623590a" (UID: "0b449cf6-1294-4b9b-9974-7553d623590a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:48.584836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.584805 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h" pod "0b449cf6-1294-4b9b-9974-7553d623590a" (UID: "0b449cf6-1294-4b9b-9974-7553d623590a"). InnerVolumeSpecName "kube-api-access-29d2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:48.585000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.584886 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca" pod "0b449cf6-1294-4b9b-9974-7553d623590a" (UID: "0b449cf6-1294-4b9b-9974-7553d623590a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:48.681422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.681379 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:48.681620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.681446 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:48.684654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.684616 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:48.684654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.684656 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b449cf6-1294-4b9b-9974-7553d623590a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:48.684940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:48.684672 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29d2h\" (UniqueName: \"kubernetes.io/projected/0b449cf6-1294-4b9b-9974-7553d623590a-kube-api-access-29d2h\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:49.305939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.305906 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" Nov 03 03:25:49.306131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.305950 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26" event={"ID":"0b449cf6-1294-4b9b-9974-7553d623590a","Type":"ContainerDied","Data":"7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2"} Nov 03 03:25:49.306131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.306024 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5d7269a78898c15f4a345503670110344c921d427276ad30332ab75a5fa5c2" Nov 03 03:25:49.309174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.309147 5275 scope.go:122] "RemoveContainer" containerID="fe58c7a5db8bf673023ce411959c8ab0d81ed7f4bd22191d17ea9a2f526ab9fb" Nov 03 03:25:49.321049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.321008 5275 scope.go:122] "RemoveContainer" containerID="d6f8496c790ea22ad097ed25983d9542882ea3b889ec28eab8203e35f5028100" Nov 03 03:25:49.333230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.333205 5275 scope.go:122] "RemoveContainer" containerID="390c7149679e046245dfb4ce83c76eb8eb4e613d84da84398e5e8612e8cd8f40" Nov 03 03:25:49.681260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.681167 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:49.681883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:49.681824 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:50.667524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:50.667479 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:25:50.667706 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:50.667552 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 03:25:51.671716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:51.671689 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:51.672254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:51.672236 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:51.672840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:51.672818 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:25:52.874271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.874170 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:25:52.874709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.874404 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" containerID="cri-o://c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869" gracePeriod=30 Nov 03 03:25:52.925928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.925882 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:25:52.926822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.926799 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0b449cf6-1294-4b9b-9974-7553d623590a" containerName="revoke-keys" Nov 03 03:25:52.926822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.926823 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b449cf6-1294-4b9b-9974-7553d623590a" containerName="revoke-keys" Nov 03 03:25:52.927022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.926868 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4ee72008-e392-48b3-aed9-53de33ebf731" containerName="revoke-keys" Nov 03 03:25:52.927022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.926876 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee72008-e392-48b3-aed9-53de33ebf731" containerName="revoke-keys" Nov 03 03:25:52.927130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.927051 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b449cf6-1294-4b9b-9974-7553d623590a" containerName="revoke-keys" Nov 03 03:25:52.927130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.927068 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee72008-e392-48b3-aed9-53de33ebf731" containerName="revoke-keys" Nov 03 03:25:52.933095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.933074 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:52.988048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.988000 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9902a0c2\"" Nov 03 03:25:52.988281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.988205 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9902a0c2-dockercfg-twl4t\"" Nov 03 03:25:52.991556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:52.991524 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:25:53.062881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.062693 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.063056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.062983 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8869\" (UniqueName: \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.159783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.153976 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb"] Nov 03 03:25:53.161272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.161027 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238c6a5f-99vsb"] Nov 03 03:25:53.165022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.164985 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8869\" (UniqueName: \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.165158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.165108 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.166244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.166216 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.169289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.169263 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:25:53.176838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.175039 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.185448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.185334 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:25:53.227450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.227420 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8869\" (UniqueName: \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.247312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.247280 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:53.267146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.267102 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtlj\" (UniqueName: \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.267435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.267408 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.368540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.368504 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.368724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.368640 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtlj\" (UniqueName: \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.369421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.369397 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.430184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.430100 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtlj\" (UniqueName: \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.494322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.494292 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:53.633615 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:53.633581 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5b5ba1_1c46_4864_991c_24174b887081.slice/crio-ae31f9aef57fbf998cbf934d1ce2d3faf1aebad531e3a46d03d2949a654013a8 WatchSource:0}: Error finding container ae31f9aef57fbf998cbf934d1ce2d3faf1aebad531e3a46d03d2949a654013a8: Status 404 returned error can't find the container with id ae31f9aef57fbf998cbf934d1ce2d3faf1aebad531e3a46d03d2949a654013a8 Nov 03 03:25:53.634023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.633998 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:25:53.876060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:53.876019 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:25:53.876431 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:25:53.876101 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fef473e_8c1d_467f_9831_c8946dfa0644.slice/crio-c8b2fde0c95ff1196ee17709ac804eddd5fa9f2cd19621d9e81176dfe454653d WatchSource:0}: Error finding container c8b2fde0c95ff1196ee17709ac804eddd5fa9f2cd19621d9e81176dfe454653d: Status 404 returned error can't find the container with id c8b2fde0c95ff1196ee17709ac804eddd5fa9f2cd19621d9e81176dfe454653d Nov 03 03:25:54.350532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.350493 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" event={"ID":"1fef473e-8c1d-467f-9831-c8946dfa0644","Type":"ContainerStarted","Data":"f092eafe27eed1bbc9e641948903ca3d3c45e6803a07161c82aa9514e262bd20"} Nov 03 03:25:54.350714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.350566 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:25:54.350714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.350622 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" event={"ID":"1fef473e-8c1d-467f-9831-c8946dfa0644","Type":"ContainerStarted","Data":"c8b2fde0c95ff1196ee17709ac804eddd5fa9f2cd19621d9e81176dfe454653d"} Nov 03 03:25:54.353028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.352997 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" event={"ID":"8b5b5ba1-1c46-4864-991c-24174b887081","Type":"ContainerStarted","Data":"e85c3d66900b75e99e1790b7c195c3aea057309b5fc44962d11351d711814256"} Nov 03 03:25:54.353193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.353047 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" event={"ID":"8b5b5ba1-1c46-4864-991c-24174b887081","Type":"ContainerStarted","Data":"ae31f9aef57fbf998cbf934d1ce2d3faf1aebad531e3a46d03d2949a654013a8"} Nov 03 03:25:54.353193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.353109 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:25:54.473353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.473293 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" podStartSLOduration=1.473278107 podStartE2EDuration="1.473278107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:54.468392414 +0000 UTC m=+1445.621080837" watchObservedRunningTime="2026-11-03 03:25:54.473278107 +0000 UTC m=+1445.625966528" Nov 03 03:25:54.637496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.637365 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" podStartSLOduration=2.637341029 podStartE2EDuration="2.637341029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:54.592133976 +0000 UTC m=+1445.744822404" watchObservedRunningTime="2026-11-03 03:25:54.637341029 +0000 UTC m=+1445.790029458" Nov 03 03:25:54.986800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:54.986720 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e7d66c-ead1-4392-8295-6bde48cfaf7a" path="/var/lib/kubelet/pods/78e7d66c-ead1-4392-8295-6bde48cfaf7a/volumes" Nov 03 03:25:56.249412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:56.249372 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:25:56.708830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:56.708789 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:25:56.710692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:56.710647 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:25:56.759665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:56.759604 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:25:58.154423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.154342 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:25:58.245190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245156 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245195 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245213 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245265 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245306 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245334 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245504 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:25:58.245653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245515 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:25:58.245653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245538 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245576 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245656 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\") pod \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\" (UID: \"15d9fa65-1bbe-44e6-8112-b3e3d7446920\") " Nov 03 03:25:58.245950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245685 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:25:58.245950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.245805 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:25:58.246107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246010 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:58.246354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246333 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.246354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246357 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.246543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246371 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.246543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246384 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.246543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.246396 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.248039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.248008 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:25:58.248039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.248019 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:25:58.248446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.248425 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "kube-api-access-4tkjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:58.248493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.248464 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token" pod "15d9fa65-1bbe-44e6-8112-b3e3d7446920" (UID: "15d9fa65-1bbe-44e6-8112-b3e3d7446920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:58.348513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.348471 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tkjw\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-kube-api-access-4tkjw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.348513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.348512 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.348759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.348528 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.348759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.348542 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d9fa65-1bbe-44e6-8112-b3e3d7446920-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:25:58.387089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.386994 5275 generic.go:356] "Generic (PLEG): container finished" podID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerID="c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869" exitCode=0 Nov 03 03:25:58.387265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.387205 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" Nov 03 03:25:58.388045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.388014 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" event={"ID":"15d9fa65-1bbe-44e6-8112-b3e3d7446920","Type":"ContainerDied","Data":"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869"} Nov 03 03:25:58.388167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.388082 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn" event={"ID":"15d9fa65-1bbe-44e6-8112-b3e3d7446920","Type":"ContainerDied","Data":"9100a6855388abdcc798d2d874ccf29c4fcac004231cdd396665d4af403ef7e0"} Nov 03 03:25:58.388167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.388114 5275 scope.go:122] "RemoveContainer" containerID="c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869" Nov 03 03:25:58.400745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.400723 5275 scope.go:122] "RemoveContainer" containerID="c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869" Nov 03 03:25:58.401161 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:25:58.401110 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869\": container with ID starting with c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869 not found: ID does not exist" containerID="c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869" Nov 03 03:25:58.401267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.401172 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869"} err="failed to get container status \"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869\": rpc error: code = NotFound desc = could not find container \"c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869\": container with ID starting with c0e216bb2ea0b9b09877bd3b7334a8d5522cd7578de5547d39ab402f1189b869 not found: ID does not exist" Nov 03 03:25:58.557370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.557331 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:25:58.563884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.563830 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-jhcjn"] Nov 03 03:25:58.988368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:25:58.988289 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" path="/var/lib/kubelet/pods/15d9fa65-1bbe-44e6-8112-b3e3d7446920/volumes" Nov 03 03:26:01.632634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.632595 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht"] Nov 03 03:26:01.633301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.633285 5275 cpu_manager.go:414] "containerMap: removing container" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" Nov 03 03:26:01.633301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.633301 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" Nov 03 03:26:01.633397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.633388 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d9fa65-1bbe-44e6-8112-b3e3d7446920" containerName="istio-proxy" Nov 03 03:26:01.637129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.637110 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.695748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.695719 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht"] Nov 03 03:26:01.710109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.710072 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" containerID="cri-o://73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277" gracePeriod=25 Nov 03 03:26:01.710953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.710925 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.711164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.711137 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tccn\" (UniqueName: \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.711267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.711215 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.738688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.738658 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:26:01.738688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.738658 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:26:01.738897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.738742 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:26:01.739825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.739799 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:26:01.760108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.760065 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" podUID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" containerName="payload-processing" containerID="cri-o://8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e" gracePeriod=25 Nov 03 03:26:01.781134 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:01.781096 5275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aaeb03_26af_4d79_99f9_28656a7dc626.slice/crio-73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aaeb03_26af_4d79_99f9_28656a7dc626.slice/crio-conmon-73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277.scope\": RecentStats: unable to find data in memory cache]" Nov 03 03:26:01.812489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.812461 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.812639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.812496 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tccn\" (UniqueName: \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.812639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.812534 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.813306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.813279 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.815148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.815123 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.873656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.873632 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tccn\" (UniqueName: \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\") pod \"maas-api-revoke-sub-9a53ca6a-98xht\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:01.954985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:01.952507 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:02.039992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.039692 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:26:02.117362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.116460 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\") pod \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " Nov 03 03:26:02.117362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.116579 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\") pod \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " Nov 03 03:26:02.117362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.116608 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\") pod \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " Nov 03 03:26:02.117362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.116666 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\") pod \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\" (UID: \"8d1a4a81-6298-448f-ad39-1f770dfcc4ac\") " Nov 03 03:26:02.118748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.118703 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume" pod "8d1a4a81-6298-448f-ad39-1f770dfcc4ac" (UID: "8d1a4a81-6298-448f-ad39-1f770dfcc4ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:02.118975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.118943 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume" pod "8d1a4a81-6298-448f-ad39-1f770dfcc4ac" (UID: "8d1a4a81-6298-448f-ad39-1f770dfcc4ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:02.121735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.121399 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm" pod "8d1a4a81-6298-448f-ad39-1f770dfcc4ac" (UID: "8d1a4a81-6298-448f-ad39-1f770dfcc4ac"). InnerVolumeSpecName "kube-api-access-glhlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:02.125130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.125101 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials" pod "8d1a4a81-6298-448f-ad39-1f770dfcc4ac" (UID: "8d1a4a81-6298-448f-ad39-1f770dfcc4ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:02.136014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.135986 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:26:02.219565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.219484 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\") pod \"08aaeb03-26af-4d79-99f9-28656a7dc626\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " Nov 03 03:26:02.219565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.219537 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\") pod \"08aaeb03-26af-4d79-99f9-28656a7dc626\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " Nov 03 03:26:02.219750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.219642 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\") pod \"08aaeb03-26af-4d79-99f9-28656a7dc626\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " Nov 03 03:26:02.219821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.219795 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\") pod \"08aaeb03-26af-4d79-99f9-28656a7dc626\" (UID: \"08aaeb03-26af-4d79-99f9-28656a7dc626\") " Nov 03 03:26:02.220556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.220531 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhlm\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-kube-api-access-glhlm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.220718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.220561 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.220718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.220576 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.220718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.220590 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1a4a81-6298-448f-ad39-1f770dfcc4ac-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.220982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.220834 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume" pod "08aaeb03-26af-4d79-99f9-28656a7dc626" (UID: "08aaeb03-26af-4d79-99f9-28656a7dc626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:02.221080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.221051 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume" pod "08aaeb03-26af-4d79-99f9-28656a7dc626" (UID: "08aaeb03-26af-4d79-99f9-28656a7dc626"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:02.222666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.222635 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials" pod "08aaeb03-26af-4d79-99f9-28656a7dc626" (UID: "08aaeb03-26af-4d79-99f9-28656a7dc626"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:02.222948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.222925 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs" pod "08aaeb03-26af-4d79-99f9-28656a7dc626" (UID: "08aaeb03-26af-4d79-99f9-28656a7dc626"). InnerVolumeSpecName "kube-api-access-l46xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:02.331871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.331430 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.331871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.331465 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.331871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.331483 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aaeb03-26af-4d79-99f9-28656a7dc626-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.331871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.331499 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46xs\" (UniqueName: \"kubernetes.io/projected/08aaeb03-26af-4d79-99f9-28656a7dc626-kube-api-access-l46xs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:02.345571 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:02.345464 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:26:02.394116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.394091 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht"] Nov 03 03:26:02.396253 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:02.396226 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3419e1_7143_4259_a4d2_6aec5866035a.slice/crio-5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54 WatchSource:0}: Error finding container 5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54: Status 404 returned error can't find the container with id 5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54 Nov 03 03:26:02.429831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.428886 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" event={"ID":"6f3419e1-7143-4259-a4d2-6aec5866035a","Type":"ContainerStarted","Data":"5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54"} Nov 03 03:26:02.430837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.430805 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" containerID="8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e" exitCode=0 Nov 03 03:26:02.431006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.430923 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" Nov 03 03:26:02.431069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.431030 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" event={"ID":"8d1a4a81-6298-448f-ad39-1f770dfcc4ac","Type":"ContainerDied","Data":"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e"} Nov 03 03:26:02.431117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.431077 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr" event={"ID":"8d1a4a81-6298-448f-ad39-1f770dfcc4ac","Type":"ContainerDied","Data":"557b2c4b58028e4c382f3ba3616516c3595feb83107e82315d85099a502d05f4"} Nov 03 03:26:02.431117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.431107 5275 scope.go:122] "RemoveContainer" containerID="8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e" Nov 03 03:26:02.435768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.435243 5275 generic.go:356] "Generic (PLEG): container finished" podID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerID="73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277" exitCode=0 Nov 03 03:26:02.435768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.435299 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" Nov 03 03:26:02.435768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.435292 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" event={"ID":"08aaeb03-26af-4d79-99f9-28656a7dc626","Type":"ContainerDied","Data":"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277"} Nov 03 03:26:02.435768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.435402 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn" event={"ID":"08aaeb03-26af-4d79-99f9-28656a7dc626","Type":"ContainerDied","Data":"2c4d802b68759c17925960d110e0d7a09c4d8d07012bbe023b00b1a32546a40e"} Nov 03 03:26:02.457252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.457220 5275 scope.go:122] "RemoveContainer" containerID="8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e" Nov 03 03:26:02.457733 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:02.457705 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e\": container with ID starting with 8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e not found: ID does not exist" containerID="8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e" Nov 03 03:26:02.457834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.457747 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e"} err="failed to get container status \"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e\": rpc error: code = NotFound desc = could not find container \"8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e\": container with ID starting with 8f82f11cb84da38ec07742a78b1bfee9873c01514df868f568a0e78c869efe6e not found: ID does not exist" Nov 03 03:26:02.457834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.457771 5275 scope.go:122] "RemoveContainer" containerID="73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277" Nov 03 03:26:02.473540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.473421 5275 scope.go:122] "RemoveContainer" containerID="73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277" Nov 03 03:26:02.474124 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:02.474037 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277\": container with ID starting with 73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277 not found: ID does not exist" containerID="73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277" Nov 03 03:26:02.474124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.474074 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277"} err="failed to get container status \"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277\": rpc error: code = NotFound desc = could not find container \"73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277\": container with ID starting with 73d355d5ed4ae956189ea9688d058777eec3a9bc1875868babdd8643e8cba277 not found: ID does not exist" Nov 03 03:26:02.604796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.604766 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:26:02.611307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.611274 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-qcjcn"] Nov 03 03:26:02.750302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.750243 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:26:02.807189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:02.807153 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d94f86b-2kpvr"] Nov 03 03:26:03.166497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:03.166451 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" path="/var/lib/kubelet/pods/08aaeb03-26af-4d79-99f9-28656a7dc626/volumes" Nov 03 03:26:03.167317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:03.167288 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" path="/var/lib/kubelet/pods/8d1a4a81-6298-448f-ad39-1f770dfcc4ac/volumes" Nov 03 03:26:03.505570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:03.502256 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" event={"ID":"6f3419e1-7143-4259-a4d2-6aec5866035a","Type":"ContainerStarted","Data":"5f0b9e1700f69447d6315ac629756470a2aa07467da2cac32b9585365353f3f8"} Nov 03 03:26:04.046875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.046017 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.394141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.391640 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.421877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.420868 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.421877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.421290 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:26:04.447873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.446939 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.610982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.610947 5275 generic.go:356] "Generic (PLEG): container finished" podID="6f3419e1-7143-4259-a4d2-6aec5866035a" containerID="5f0b9e1700f69447d6315ac629756470a2aa07467da2cac32b9585365353f3f8" exitCode=0 Nov 03 03:26:04.611510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.611485 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" event={"ID":"6f3419e1-7143-4259-a4d2-6aec5866035a","Type":"ContainerDied","Data":"5f0b9e1700f69447d6315ac629756470a2aa07467da2cac32b9585365353f3f8"} Nov 03 03:26:04.885903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.869902 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.930295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.929947 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:04.930295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:04.930022 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.058981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.058038 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.078888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.071171 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:26:05.184492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.183637 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.409579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.409519 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.411300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.411278 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:26:05.436464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.436348 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.436960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.436917 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.438129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.438110 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:26:05.443134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.443115 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:26:05.469140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.447179 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.470319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.470283 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:26:05.489927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.478055 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:26:05.806871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.801968 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:05.847641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.832928 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.874250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.863715 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:26:05.889285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.889242 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:26:05.891360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.891332 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:26:05.922402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:05.922360 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.083936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.079535 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.299873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.296296 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:06.311873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.307936 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.343060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.342956 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.460357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.434026 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:06.490606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.442950 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.492293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.443001 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.492293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.491272 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.797874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.797091 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.965226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:06.965180 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:26:07.208213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.009501 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:07.208213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.206010 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:07.208213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.168098 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 03:26:07.208213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.200953 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:07.208213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.206366 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:07.248242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.241022 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:07.310128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.304049 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 03 03:26:07.578886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.578201 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:26:07.582877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.579252 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:07.652408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.652346 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:26:07.973894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.971468 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:07.973894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:07.971549 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:08.077044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.076990 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:26:08.254903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.254779 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:26:08.331874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.285325 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:26:08.358872 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:08.299688 5275 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.323s" Nov 03 03:26:08.358872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.358456 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:26:08.359133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.310373 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 03:26:08.582967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.582919 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:26:08.716875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.582832 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:08.805921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.780525 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.790446 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:08.858053 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283023 5275 cpu_manager.go:414] "containerMap: removing container" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283043 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283107 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" containerName="payload-processing" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283116 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" containerName="payload-processing" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283276 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1a4a81-6298-448f-ad39-1f770dfcc4ac" containerName="payload-processing" Nov 03 03:26:09.291221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.283294 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="08aaeb03-26af-4d79-99f9-28656a7dc626" containerName="payload-processing" Nov 03 03:26:09.322546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.135929 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.329165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.263400 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.361884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.359985 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.362130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.358399 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.392089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.281445 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.561111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.557978 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:09.565459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.565407 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.596672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.596629 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.596955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.596931 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.597191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.597172 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.597334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.597318 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.597443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.597429 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.597653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.597637 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.597789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.597775 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bcx\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.598203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.598172 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.624321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.624289 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 03:26:09.625995 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.625972 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 03:26:09.641220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.640539 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4d77v\"" Nov 03 03:26:09.659876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.651656 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" event={"ID":"6f3419e1-7143-4259-a4d2-6aec5866035a","Type":"ContainerDied","Data":"5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54"} Nov 03 03:26:09.660140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.660117 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8103064a3105e63c5c1d39445b380e3d72a7db111775fb28d98fe766558c54" Nov 03 03:26:09.660365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.660350 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.701416 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\") pod \"6f3419e1-7143-4259-a4d2-6aec5866035a\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.701654 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\") pod \"6f3419e1-7143-4259-a4d2-6aec5866035a\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.701918 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\" (UniqueName: \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\") pod \"6f3419e1-7143-4259-a4d2-6aec5866035a\" (UID: \"6f3419e1-7143-4259-a4d2-6aec5866035a\") " Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702298 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702368 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702423 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702471 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702518 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702669 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.702737 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bcx\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.725514 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.726870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.725837 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca" pod "6f3419e1-7143-4259-a4d2-6aec5866035a" (UID: "6f3419e1-7143-4259-a4d2-6aec5866035a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:09.728481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.727779 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:09.745097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.743996 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.745497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.745473 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 03:26:09.883086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.760053 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn" pod "6f3419e1-7143-4259-a4d2-6aec5866035a" (UID: "6f3419e1-7143-4259-a4d2-6aec5866035a"). InnerVolumeSpecName "kube-api-access-6tccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:09.883323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.882959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca" pod "6f3419e1-7143-4259-a4d2-6aec5866035a" (UID: "6f3419e1-7143-4259-a4d2-6aec5866035a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:26:09.883468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.827538 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:09.883579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.763114 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.933935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.933902 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.937805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.937781 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:26:09.942780 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.942747 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bcx\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.943339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.943314 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.946344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.946312 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2rwq6\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.967081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.966722 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:09.994277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.994237 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:26:09.996108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.996080 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tccn\" (UniqueName: \"kubernetes.io/projected/6f3419e1-7143-4259-a4d2-6aec5866035a-kube-api-access-6tccn\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:09.996275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:09.996260 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3419e1-7143-4259-a4d2-6aec5866035a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:10.022441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.022411 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:26:10.128632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.128603 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:26:10.185117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.185031 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:26:10.197868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.186219 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:26:10.197868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.197868 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:26:10.379754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.379727 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht" Nov 03 03:26:10.596183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.596152 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:26:10.597270 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:10.597240 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd59738_5074_4422_913f_1d9e55ead178.slice/crio-bdab0d926dd192197270d6d9b751d05a59ae3ef02fd7b2a048000826fc1c0552 WatchSource:0}: Error finding container bdab0d926dd192197270d6d9b751d05a59ae3ef02fd7b2a048000826fc1c0552: Status 404 returned error can't find the container with id bdab0d926dd192197270d6d9b751d05a59ae3ef02fd7b2a048000826fc1c0552 Nov 03 03:26:10.599945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:10.599918 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:26:11.069732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.069693 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd"] Nov 03 03:26:11.074181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.074148 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6f3419e1-7143-4259-a4d2-6aec5866035a" containerName="revoke-keys" Nov 03 03:26:11.074405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.074383 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3419e1-7143-4259-a4d2-6aec5866035a" containerName="revoke-keys" Nov 03 03:26:11.074738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.074718 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3419e1-7143-4259-a4d2-6aec5866035a" containerName="revoke-keys" Nov 03 03:26:11.083908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.082622 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.131146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.131117 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd"] Nov 03 03:26:11.136891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.135785 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:26:11.136891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.135918 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:26:11.136891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.135792 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:26:11.136891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.135929 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:26:11.141626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.139940 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzr7\" (UniqueName: \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.141626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.140063 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.141626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.140181 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.241904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.241860 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzr7\" (UniqueName: \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.242096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.241936 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.242096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.242043 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.242914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.242860 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.244768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.244742 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.303237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.303203 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzr7\" (UniqueName: \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\") pod \"maas-api-revoke-sub-f25dca97-crlkd\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.391102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.391019 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" event={"ID":"cbd59738-5074-4422-913f-1d9e55ead178","Type":"ContainerStarted","Data":"bdab0d926dd192197270d6d9b751d05a59ae3ef02fd7b2a048000826fc1c0552"} Nov 03 03:26:11.405371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.405343 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:11.828078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:11.827749 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd"] Nov 03 03:26:11.829069 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:11.828418 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2e7a3e_7f69_448c_ab1f_f555ec6c83c2.slice/crio-67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0 WatchSource:0}: Error finding container 67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0: Status 404 returned error can't find the container with id 67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0 Nov 03 03:26:12.402527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:12.402490 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" event={"ID":"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2","Type":"ContainerStarted","Data":"f82bd16419b5254658e79a0761209f461451d68fd435770c3aafed14bb30a806"} Nov 03 03:26:12.402969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:12.402550 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" event={"ID":"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2","Type":"ContainerStarted","Data":"67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0"} Nov 03 03:26:13.415268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:13.415220 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" containerID="f82bd16419b5254658e79a0761209f461451d68fd435770c3aafed14bb30a806" exitCode=0 Nov 03 03:26:13.415677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:13.415440 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" event={"ID":"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2","Type":"ContainerDied","Data":"f82bd16419b5254658e79a0761209f461451d68fd435770c3aafed14bb30a806"} Nov 03 03:26:14.634866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.634819 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:14.721641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.721540 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\" (UniqueName: \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\") pod \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " Nov 03 03:26:14.721641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.721627 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\") pod \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " Nov 03 03:26:14.721877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.721777 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\") pod \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\" (UID: \"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2\") " Nov 03 03:26:14.722173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.722140 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca" pod "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" (UID: "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:14.722479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.722449 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:14.724736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.724708 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca" pod "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" (UID: "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:26:14.724977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.724948 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7" pod "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" (UID: "8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2"). InnerVolumeSpecName "kube-api-access-xnzr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:14.823746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.823709 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzr7\" (UniqueName: \"kubernetes.io/projected/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-kube-api-access-xnzr7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:14.823746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:14.823746 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:15.435021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:15.434985 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" event={"ID":"8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2","Type":"ContainerDied","Data":"67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0"} Nov 03 03:26:15.435165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:15.435033 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd" Nov 03 03:26:15.435165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:15.435047 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67870ce27ff0bdc8d62b33acf2bf18efd01b44aaa416b637b3b1da0520d89ab0" Nov 03 03:26:15.482918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:15.482885 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:26:15.483029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:15.482950 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:26:16.446814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:16.446772 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" event={"ID":"cbd59738-5074-4422-913f-1d9e55ead178","Type":"ContainerStarted","Data":"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e"} Nov 03 03:26:16.447285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:16.447004 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:16.449154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:16.449123 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2rwq6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:26:16.449305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:16.449176 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" podUID="cbd59738-5074-4422-913f-1d9e55ead178" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:26:16.661606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:16.660047 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" podStartSLOduration=4.778795611 podStartE2EDuration="9.65998827s" totalImagesPullingTime="4.881192659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:16.619619147 +0000 UTC m=+1467.772307578" watchObservedRunningTime="2026-11-03 03:26:16.65998827 +0000 UTC m=+1467.812676692" Nov 03 03:26:17.461136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:17.461104 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:26:19.709347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.709317 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:26:19.709946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.709931 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" containerName="revoke-keys" Nov 03 03:26:19.710003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.709946 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" containerName="revoke-keys" Nov 03 03:26:19.710079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.710069 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" containerName="revoke-keys" Nov 03 03:26:19.715184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.715165 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908092 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908137 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908229 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908438 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908459 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rqr\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908512 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.908611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.908596 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:19.927940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:19.927905 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:26:20.009909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.009820 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.009909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.009906 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.009956 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010048 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010091 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010142 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010169 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rqr\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010196 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010325 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010449 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010537 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010626 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.010778 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.010759 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.011329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.011309 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.012552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.012531 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.012671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.012653 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.071211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.071172 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.071361 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.071261 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rqr\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.329337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.329297 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:20.765631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.765604 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:26:20.766490 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:20.766455 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf073012_438e_48f7_b1b4_d5230919e381.slice/crio-41d3a3b9643e549ae87bc1b8df0d2efb9f81036970a2dcb7319d5b566b636e37 WatchSource:0}: Error finding container 41d3a3b9643e549ae87bc1b8df0d2efb9f81036970a2dcb7319d5b566b636e37: Status 404 returned error can't find the container with id 41d3a3b9643e549ae87bc1b8df0d2efb9f81036970a2dcb7319d5b566b636e37 Nov 03 03:26:20.769186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.769148 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:26:20.769273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.769241 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:26:20.769312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:20.769287 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:26:21.502947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:21.502916 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" event={"ID":"bf073012-438e-48f7-b1b4-d5230919e381","Type":"ContainerStarted","Data":"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf"} Nov 03 03:26:21.503119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:21.502965 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" event={"ID":"bf073012-438e-48f7-b1b4-d5230919e381","Type":"ContainerStarted","Data":"41d3a3b9643e549ae87bc1b8df0d2efb9f81036970a2dcb7319d5b566b636e37"} Nov 03 03:26:21.680186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:21.680129 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podStartSLOduration=2.680114908 podStartE2EDuration="2.680114908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:21.676136516 +0000 UTC m=+1472.828824952" watchObservedRunningTime="2026-11-03 03:26:21.680114908 +0000 UTC m=+1472.832803328" Nov 03 03:26:21.996422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:21.996396 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8"] Nov 03 03:26:22.002190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.002162 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8"] Nov 03 03:26:22.002397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.002271 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.046641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.046603 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.046820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.046756 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.046985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.046960 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tw6m\" (UniqueName: \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.076333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.075965 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:26:22.076333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.076049 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:26:22.076333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.076119 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:26:22.076664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.076486 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:26:22.148143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.148110 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.148288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.148231 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.148355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.148345 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tw6m\" (UniqueName: \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.148909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.148887 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.165211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.151248 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.209569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.209542 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tw6m\" (UniqueName: \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\") pod \"maas-api-revoke-sub-2de8f982-92wr8\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.334335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.334292 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:22.334655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.334640 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:22.336033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.336003 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:22.336156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.336057 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:22.741837 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:22.741809 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fd9546_bdd6_4e59_a030_2115031e8e25.slice/crio-a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9 WatchSource:0}: Error finding container a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9: Status 404 returned error can't find the container with id a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9 Nov 03 03:26:22.742717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:22.742691 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8"] Nov 03 03:26:23.330124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:23.330093 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:23.330502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:23.330147 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:23.526717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:23.526682 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" event={"ID":"37fd9546-bdd6-4e59-a030-2115031e8e25","Type":"ContainerStarted","Data":"4e2d577a068e1abdb8a29cafaf45b1e9429aab3abd2e4839183b696a08c563b6"} Nov 03 03:26:23.526925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:23.526735 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" event={"ID":"37fd9546-bdd6-4e59-a030-2115031e8e25","Type":"ContainerStarted","Data":"a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9"} Nov 03 03:26:23.645268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:23.645202 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" podStartSLOduration=2.6451881459999997 podStartE2EDuration="2.645188146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:23.641923602 +0000 UTC m=+1474.794612024" watchObservedRunningTime="2026-11-03 03:26:23.645188146 +0000 UTC m=+1474.797876566" Nov 03 03:26:24.330615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:24.330580 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:24.331062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:24.330634 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:24.536588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:24.536555 5275 generic.go:356] "Generic (PLEG): container finished" podID="37fd9546-bdd6-4e59-a030-2115031e8e25" containerID="4e2d577a068e1abdb8a29cafaf45b1e9429aab3abd2e4839183b696a08c563b6" exitCode=0 Nov 03 03:26:24.536737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:24.536602 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" event={"ID":"37fd9546-bdd6-4e59-a030-2115031e8e25","Type":"ContainerDied","Data":"4e2d577a068e1abdb8a29cafaf45b1e9429aab3abd2e4839183b696a08c563b6"} Nov 03 03:26:25.330165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.330134 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:25.330383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.330362 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:25.782035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.782006 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:25.830189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.830160 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\") pod \"37fd9546-bdd6-4e59-a030-2115031e8e25\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " Nov 03 03:26:25.830332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.830238 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\" (UniqueName: \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\") pod \"37fd9546-bdd6-4e59-a030-2115031e8e25\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " Nov 03 03:26:25.830441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.830410 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\") pod \"37fd9546-bdd6-4e59-a030-2115031e8e25\" (UID: \"37fd9546-bdd6-4e59-a030-2115031e8e25\") " Nov 03 03:26:25.830724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.830672 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca" pod "37fd9546-bdd6-4e59-a030-2115031e8e25" (UID: "37fd9546-bdd6-4e59-a030-2115031e8e25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:25.831106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.831039 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:25.832569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.832546 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca" pod "37fd9546-bdd6-4e59-a030-2115031e8e25" (UID: "37fd9546-bdd6-4e59-a030-2115031e8e25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:26:25.832791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.832764 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m" pod "37fd9546-bdd6-4e59-a030-2115031e8e25" (UID: "37fd9546-bdd6-4e59-a030-2115031e8e25"). InnerVolumeSpecName "kube-api-access-9tw6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:25.945053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.941595 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tw6m\" (UniqueName: \"kubernetes.io/projected/37fd9546-bdd6-4e59-a030-2115031e8e25-kube-api-access-9tw6m\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:25.945053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:25.941637 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fd9546-bdd6-4e59-a030-2115031e8e25-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:26.329760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:26.329726 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:26.329988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:26.329787 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:26.577161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:26.577130 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" Nov 03 03:26:26.577331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:26.577164 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8" event={"ID":"37fd9546-bdd6-4e59-a030-2115031e8e25","Type":"ContainerDied","Data":"a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9"} Nov 03 03:26:26.577331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:26.577225 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e50c61bc69fcd2af0606f4bafcda17167ea055cf62146518196778562318c9" Nov 03 03:26:27.329989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:27.329955 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 03:26:27.330455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:27.330018 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 03:26:28.334397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:28.334362 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:28.335240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:28.335214 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:28.336630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:28.336613 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:26:32.330965 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:32.330929 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:26:44.511351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:44.511303 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:44.540274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:44.540161 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.084136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.084083 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.092690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.092639 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.151265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.150887 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.349926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.328656 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.349926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.328977 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:26:45.541643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.538922 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.541643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.539617 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:26:45.917884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:45.915079 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.176927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.137186 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.179353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.141598 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.184872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.180965 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:26:46.202585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.185137 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:26:46.207872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.151166 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.234905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.208166 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:26:46.322912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.318655 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.329617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.329562 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.371905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.354442 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.371905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.356873 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:26:46.371905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.356943 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:26:46.551321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.549959 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.848958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:46.837485 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:47.207272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.207164 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:26:47.208265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.208231 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 03:26:47.227001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.226960 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:26:47.341745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.341706 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 03 03:26:47.347999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.342513 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:47.415873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.413409 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:26:47.553578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.551123 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:47.631980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:47.631936 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:26:48.228907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.227955 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.228907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.228761 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.229189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.229020 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:26:48.359112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.359062 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.447873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.435892 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.590927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.576713 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:48.629989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.620931 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6948b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:48.630326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.630272 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.664996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.664945 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:48.665682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.665658 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:26:48.804011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:48.803979 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:26:49.309874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.243047 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:26:49.359497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.340233 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:26:49.359497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.340296 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:26:49.359497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.348418 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:26:49.359497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.351393 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:26:49.359497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.358549 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:26:49.365871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.363096 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:26:49.413469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.413440 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:26:49.414118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.414087 5275 scope.go:122] "RemoveContainer" containerID="70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07" Nov 03 03:26:49.432391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.432362 5275 scope.go:122] "RemoveContainer" containerID="8c4c94a784417367dfe3ac143772315be1463efd43e1e2b283d1cd445dbe2c04" Nov 03 03:26:49.440763 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:49.440633 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07\": container with ID starting with 70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07 not found: ID does not exist" containerID="70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07" Nov 03 03:26:49.440997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.440972 5275 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07\": container with ID starting with 70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07 not found: ID does not exist" Nov 03 03:26:49.443925 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:26:49.443899 5275 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07\": container with ID starting with 70583c8de122cec85e6eca48b4b64b3c923398bce1336e2d9e8e67b5dce8da07 not found: ID does not exist" Nov 03 03:26:49.456138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.456114 5275 scope.go:122] "RemoveContainer" containerID="3eb4e593ae091070482a9a56cc29f98803e99558f43a9090c0ea85a83572e32c" Nov 03 03:26:49.478251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.478226 5275 scope.go:122] "RemoveContainer" containerID="fabddb54d09031992de79df675e932bc0abc8d10926d5aa1c638db2c2d55b62c" Nov 03 03:26:49.502986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:49.502960 5275 scope.go:122] "RemoveContainer" containerID="7b41b9d167edc8d5ebe9d67e6ddc6ef88172843454b41f2eec33e6ef270bbb5c" Nov 03 03:26:53.709591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.709554 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw"] Nov 03 03:26:53.710710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.710677 5275 cpu_manager.go:414] "containerMap: removing container" podUID="37fd9546-bdd6-4e59-a030-2115031e8e25" containerName="revoke-keys" Nov 03 03:26:53.710710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.710705 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fd9546-bdd6-4e59-a030-2115031e8e25" containerName="revoke-keys" Nov 03 03:26:53.710923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.710835 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fd9546-bdd6-4e59-a030-2115031e8e25" containerName="revoke-keys" Nov 03 03:26:53.719988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.719965 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.773323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.773287 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw"] Nov 03 03:26:53.797397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.797350 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xtb\" (UniqueName: \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.797566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.797426 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.797672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.797642 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.815733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.815698 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:26:53.815925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.815790 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:26:53.815925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.815790 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:26:53.815925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.815826 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:26:53.898853 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.898820 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.899042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.898978 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xtb\" (UniqueName: \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.899042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.899021 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.899463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.899442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.901544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.901522 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:53.960151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:53.960074 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xtb\" (UniqueName: \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\") pod \"maas-api-revoke-sub-5085d947-xd7zw\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:54.037154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:54.037124 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:54.419352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:54.419321 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw"] Nov 03 03:26:54.423037 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:26:54.423006 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d34f5d_ad56_46a3_82cd_9201e72435e0.slice/crio-7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda WatchSource:0}: Error finding container 7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda: Status 404 returned error can't find the container with id 7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda Nov 03 03:26:55.236990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:55.236952 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" event={"ID":"f7d34f5d-ad56-46a3-82cd-9201e72435e0","Type":"ContainerStarted","Data":"e2506994dff4a0020415cc4563cfd5094689d05479bd14a642518c4dc49e757d"} Nov 03 03:26:55.237376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:55.237014 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" event={"ID":"f7d34f5d-ad56-46a3-82cd-9201e72435e0","Type":"ContainerStarted","Data":"7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda"} Nov 03 03:26:55.353078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:55.353021 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" podStartSLOduration=2.353003747 podStartE2EDuration="2.353003747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:55.352005469 +0000 UTC m=+1506.504693890" watchObservedRunningTime="2026-11-03 03:26:55.353003747 +0000 UTC m=+1506.505692169" Nov 03 03:26:56.247875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:56.247829 5275 generic.go:356] "Generic (PLEG): container finished" podID="f7d34f5d-ad56-46a3-82cd-9201e72435e0" containerID="e2506994dff4a0020415cc4563cfd5094689d05479bd14a642518c4dc49e757d" exitCode=0 Nov 03 03:26:56.248259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:56.247889 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" event={"ID":"f7d34f5d-ad56-46a3-82cd-9201e72435e0","Type":"ContainerDied","Data":"e2506994dff4a0020415cc4563cfd5094689d05479bd14a642518c4dc49e757d"} Nov 03 03:26:57.441497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.441470 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:57.544187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.544155 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\") pod \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " Nov 03 03:26:57.544372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.544206 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\" (UniqueName: \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\") pod \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " Nov 03 03:26:57.544444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.544426 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\") pod \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\" (UID: \"f7d34f5d-ad56-46a3-82cd-9201e72435e0\") " Nov 03 03:26:57.544781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.544757 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca" pod "f7d34f5d-ad56-46a3-82cd-9201e72435e0" (UID: "f7d34f5d-ad56-46a3-82cd-9201e72435e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:57.544994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.544974 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:57.546494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.546469 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb" pod "f7d34f5d-ad56-46a3-82cd-9201e72435e0" (UID: "f7d34f5d-ad56-46a3-82cd-9201e72435e0"). InnerVolumeSpecName "kube-api-access-99xtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:57.546575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.546510 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca" pod "f7d34f5d-ad56-46a3-82cd-9201e72435e0" (UID: "f7d34f5d-ad56-46a3-82cd-9201e72435e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:26:57.646374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.646336 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d34f5d-ad56-46a3-82cd-9201e72435e0-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:57.646374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:57.646368 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xtb\" (UniqueName: \"kubernetes.io/projected/f7d34f5d-ad56-46a3-82cd-9201e72435e0-kube-api-access-99xtb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:26:58.267110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:58.267073 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" event={"ID":"f7d34f5d-ad56-46a3-82cd-9201e72435e0","Type":"ContainerDied","Data":"7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda"} Nov 03 03:26:58.267110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:58.267102 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw" Nov 03 03:26:58.267333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:26:58.267135 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7404a58c3a51f6586f66b6c12ab529681de9d73e862d8a932f0edf96caa4dcda" Nov 03 03:27:02.329306 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:27:02.329269 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:27:06.680585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.680551 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:27:06.682955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.681160 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f7d34f5d-ad56-46a3-82cd-9201e72435e0" containerName="revoke-keys" Nov 03 03:27:06.682955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.681172 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d34f5d-ad56-46a3-82cd-9201e72435e0" containerName="revoke-keys" Nov 03 03:27:06.682955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.681309 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d34f5d-ad56-46a3-82cd-9201e72435e0" containerName="revoke-keys" Nov 03 03:27:06.684784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.684764 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.767873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.745875 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b562f5-openshift-default-dockercfg-7cppb\"" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829012 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829076 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829123 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829157 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829226 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829253 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829286 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829325 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5n6b\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.841906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.829363 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.872879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.870371 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:27:06.931583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.931472 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.931763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.931664 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.931763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.931717 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.931934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.931762 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932024 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932049 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932119 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932163 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932219 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932269 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5n6b\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932324 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932559 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932164 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.932636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.932598 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.935042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.935013 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.935539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.935514 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.993394 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.993359 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:06.993584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:06.993469 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5n6b\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\") pod \"e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:07.000781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:07.000566 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:07.465636 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:27:07.465596 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7aa980_d3fc_4024_b511_642e6f7ff719.slice/crio-c5b7d29bc30cf5c8a0dfac2d93e8cc45eede71f00abf4301463a0b63d9973c22 WatchSource:0}: Error finding container c5b7d29bc30cf5c8a0dfac2d93e8cc45eede71f00abf4301463a0b63d9973c22: Status 404 returned error can't find the container with id c5b7d29bc30cf5c8a0dfac2d93e8cc45eede71f00abf4301463a0b63d9973c22 Nov 03 03:27:07.465983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:07.465956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:27:07.468335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:07.468289 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:07.468423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:07.468386 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:07.468472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:07.468432 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:08.351946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:08.351901 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerStarted","Data":"26f156c0ed23e5940068c5e45401c871f99e58ac04fc7ed615c6aaf8f09ca2fa"} Nov 03 03:27:08.352622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:08.352588 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerStarted","Data":"c5b7d29bc30cf5c8a0dfac2d93e8cc45eede71f00abf4301463a0b63d9973c22"} Nov 03 03:27:08.479887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:08.479802 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podStartSLOduration=2.479774736 podStartE2EDuration="2.479774736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:08.473158276 +0000 UTC m=+1519.625846699" watchObservedRunningTime="2026-11-03 03:27:08.479774736 +0000 UTC m=+1519.632463158" Nov 03 03:27:09.001300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:09.001259 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:09.007825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:09.007789 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:09.363979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:09.363947 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:09.364794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:09.364772 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:27:32.331347 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:27:32.331254 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:27:37.878237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:37.878197 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:27:37.883738 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:37.883709 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:37.936179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:37.936153 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\"" Nov 03 03:27:37.990833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:37.990800 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:27:38.009456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.009414 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.009631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.009558 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf42s\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.009720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.009700 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.009838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.009821 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.009943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.009918 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.010039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.010021 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.010086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.010074 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.010120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.010102 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.010179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.010161 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111119 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111167 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111210 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111234 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111278 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111373 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111465 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf42s\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111590 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111630 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111683 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.111755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111702 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.112030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.111891 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.112367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.112340 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.114130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.114107 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.114428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.114412 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.180515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.180444 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.180680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.180657 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf42s\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.199615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.199579 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:38.647581 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:27:38.647550 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea8dfcb_de12_4156_a668_d94a72673b92.slice/crio-f351e3ff62b987aa2950d5a895c6485b47aa1df31cf86f8dd91ebff35ed2152f WatchSource:0}: Error finding container f351e3ff62b987aa2950d5a895c6485b47aa1df31cf86f8dd91ebff35ed2152f: Status 404 returned error can't find the container with id f351e3ff62b987aa2950d5a895c6485b47aa1df31cf86f8dd91ebff35ed2152f Nov 03 03:27:38.647796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.647766 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:27:38.650414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.650377 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:38.650508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.650490 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:38.650548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:38.650536 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:27:39.619469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:39.619424 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" event={"ID":"dea8dfcb-de12-4156-a668-d94a72673b92","Type":"ContainerStarted","Data":"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925"} Nov 03 03:27:39.619920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:39.619527 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" event={"ID":"dea8dfcb-de12-4156-a668-d94a72673b92","Type":"ContainerStarted","Data":"f351e3ff62b987aa2950d5a895c6485b47aa1df31cf86f8dd91ebff35ed2152f"} Nov 03 03:27:39.792899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:39.792824 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podStartSLOduration=2.792808286 podStartE2EDuration="2.792808286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:39.789435052 +0000 UTC m=+1550.942123490" watchObservedRunningTime="2026-11-03 03:27:39.792808286 +0000 UTC m=+1550.945496707" Nov 03 03:27:40.199808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:40.199761 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:40.201020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:40.200992 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:40.201230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:40.201206 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:41.200835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:41.200797 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:41.201329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:41.200879 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:42.200484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:42.200442 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:42.200708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:42.200506 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:43.200321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.200237 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:43.201066 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.201032 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:43.343216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.343180 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5"] Nov 03 03:27:43.347808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.347776 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.403316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.403285 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:27:43.403823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.403606 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:27:43.404011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.403778 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:27:43.406369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.406345 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5"] Nov 03 03:27:43.448241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.448212 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:27:43.525695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.525669 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.525894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.525752 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cw4p\" (UniqueName: \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.525968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.525905 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.628466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.628417 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.628666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.628553 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cw4p\" (UniqueName: \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.628666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.628624 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.629282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.629256 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.631081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.631060 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.689908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.689875 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cw4p\" (UniqueName: \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\") pod \"maas-api-revoke-sub-4b583a92-lgrd5\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:43.695671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:43.695644 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:44.230920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:44.214196 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:44.230920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:44.214289 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:44.235876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:44.233547 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5"] Nov 03 03:27:44.793057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:44.785087 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" event={"ID":"5cd39aff-d32e-433b-8e2e-75040e135c1f","Type":"ContainerStarted","Data":"2bf5f31c32e6a486f21f04b94ea6d80e32554d975bf64bf96623a9682fcf0ff2"} Nov 03 03:27:45.248936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:45.243226 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 03:27:45.248936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:45.243285 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 03:27:45.797399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:45.797353 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" event={"ID":"5cd39aff-d32e-433b-8e2e-75040e135c1f","Type":"ContainerStarted","Data":"e764b42cf458c564f50850daa9d0b9da8cddfbff04ba957fe295b204062e5f9c"} Nov 03 03:27:45.919043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:45.918975 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" podStartSLOduration=2.918954398 podStartE2EDuration="2.918954398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:45.914556073 +0000 UTC m=+1557.067244509" watchObservedRunningTime="2026-11-03 03:27:45.918954398 +0000 UTC m=+1557.071642819" Nov 03 03:27:46.616972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:46.616944 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:46.617615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:46.617486 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:46.618490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:46.618470 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:27:46.808225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:46.808187 5275 generic.go:356] "Generic (PLEG): container finished" podID="5cd39aff-d32e-433b-8e2e-75040e135c1f" containerID="e764b42cf458c564f50850daa9d0b9da8cddfbff04ba957fe295b204062e5f9c" exitCode=0 Nov 03 03:27:46.808392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:46.808265 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" event={"ID":"5cd39aff-d32e-433b-8e2e-75040e135c1f","Type":"ContainerDied","Data":"e764b42cf458c564f50850daa9d0b9da8cddfbff04ba957fe295b204062e5f9c"} Nov 03 03:27:48.010839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.010813 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:48.121750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.121662 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\") pod \"5cd39aff-d32e-433b-8e2e-75040e135c1f\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " Nov 03 03:27:48.121750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.121744 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\") pod \"5cd39aff-d32e-433b-8e2e-75040e135c1f\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " Nov 03 03:27:48.122001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.121773 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\" (UniqueName: \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\") pod \"5cd39aff-d32e-433b-8e2e-75040e135c1f\" (UID: \"5cd39aff-d32e-433b-8e2e-75040e135c1f\") " Nov 03 03:27:48.122116 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.122087 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca" pod "5cd39aff-d32e-433b-8e2e-75040e135c1f" (UID: "5cd39aff-d32e-433b-8e2e-75040e135c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:27:48.122648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.122623 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:27:48.125238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.125207 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca" pod "5cd39aff-d32e-433b-8e2e-75040e135c1f" (UID: "5cd39aff-d32e-433b-8e2e-75040e135c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:27:48.125238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.125210 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p" pod "5cd39aff-d32e-433b-8e2e-75040e135c1f" (UID: "5cd39aff-d32e-433b-8e2e-75040e135c1f"). InnerVolumeSpecName "kube-api-access-2cw4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:27:48.224426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.224391 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd39aff-d32e-433b-8e2e-75040e135c1f-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:27:48.224426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.224423 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cw4p\" (UniqueName: \"kubernetes.io/projected/5cd39aff-d32e-433b-8e2e-75040e135c1f-kube-api-access-2cw4p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:27:48.832157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.832126 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" Nov 03 03:27:48.832157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.832141 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5" event={"ID":"5cd39aff-d32e-433b-8e2e-75040e135c1f","Type":"ContainerDied","Data":"2bf5f31c32e6a486f21f04b94ea6d80e32554d975bf64bf96623a9682fcf0ff2"} Nov 03 03:27:48.832382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:27:48.832199 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf5f31c32e6a486f21f04b94ea6d80e32554d975bf64bf96623a9682fcf0ff2" Nov 03 03:28:02.329948 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:28:02.329910 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:28:12.217359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.216981 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:28:12.241874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.233689 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:12.354883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.353935 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:28:12.378875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.368922 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:28:12.437008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.436958 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:12.807022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.806970 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:12.964874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:12.940992 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:13.228925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.226281 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:28:13.228925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.227067 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:28:13.241912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.238428 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:28:13.241912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.238893 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:28:13.245886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.243926 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:13.344185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.344156 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:28:13.357914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.357871 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:13.369566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.358720 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:28:13.845083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.820256 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:13.864924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:13.845493 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:28:14.239975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.239861 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:14.314910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.245995 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:14.314910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.246314 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:28:14.380020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.366647 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:14.482008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.470246 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:14.482008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.479909 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:28:14.486788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.486661 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:14.669828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.669789 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:28:14.832890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.832839 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:28:14.984930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:14.960223 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:28:15.097571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.097539 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:28:15.104667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.104637 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5cd39aff-d32e-433b-8e2e-75040e135c1f" containerName="revoke-keys" Nov 03 03:28:15.104902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.104882 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd39aff-d32e-433b-8e2e-75040e135c1f" containerName="revoke-keys" Nov 03 03:28:15.105267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.105248 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd39aff-d32e-433b-8e2e-75040e135c1f" containerName="revoke-keys" Nov 03 03:28:15.228884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.217067 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.243010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.242673 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-skdrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:28:15.243806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.243758 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:15.300885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.271970 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:28:15.300885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.276075 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:28:15.327501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.322314 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9902a0c2-openshift-default-dockercfg-q88mb\"" Nov 03 03:28:15.343685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.343608 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 03:28:15.531286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.531247 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541470 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541539 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541601 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541649 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541674 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q88f\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541704 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.541987 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.545209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.542044 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.643739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.643697 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.643993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.643750 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.643993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.643894 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q88f\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.643993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.643934 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.644201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644181 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.644256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644243 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.644333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644312 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.644391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644359 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.644446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644420 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.645083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644866 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.645083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644996 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.645083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.645035 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.645083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.644998 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.645389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.645116 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.646884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.646835 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.647004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.646945 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.715872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.715822 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.716099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.715986 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q88f\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:15.912031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:15.910673 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:16.436704 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:28:16.436668 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77620089_d22c_4dbc_9916_d32bd6dde877.slice/crio-6612a359488dee2efb8cde76d8b56f1778b5f047f8138faad178f71e662bf753 WatchSource:0}: Error finding container 6612a359488dee2efb8cde76d8b56f1778b5f047f8138faad178f71e662bf753: Status 404 returned error can't find the container with id 6612a359488dee2efb8cde76d8b56f1778b5f047f8138faad178f71e662bf753 Nov 03 03:28:16.437567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.437520 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:28:16.439663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.439627 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:16.439781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.439694 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:16.439781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.439726 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:16.573283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.572469 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" event={"ID":"77620089-d22c-4dbc-9916-d32bd6dde877","Type":"ContainerStarted","Data":"bd7312756456d39d8e6975792585c456bafcbcb39f14a19fbf65f45f188d73be"} Nov 03 03:28:16.573283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.572532 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" event={"ID":"77620089-d22c-4dbc-9916-d32bd6dde877","Type":"ContainerStarted","Data":"6612a359488dee2efb8cde76d8b56f1778b5f047f8138faad178f71e662bf753"} Nov 03 03:28:16.698534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.698389 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podStartSLOduration=2.698374016 podStartE2EDuration="2.698374016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:16.692331387 +0000 UTC m=+1587.845019811" watchObservedRunningTime="2026-11-03 03:28:16.698374016 +0000 UTC m=+1587.851062437" Nov 03 03:28:16.911753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:16.911725 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:17.912694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:17.912661 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:17.913151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:17.912715 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:18.911821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:18.911776 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:18.912016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:18.911880 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:19.911701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:19.911658 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:19.912257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:19.911711 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:20.911319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:20.911285 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:20.911504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:20.911343 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:21.934639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:21.934425 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:21.934639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:21.934509 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:22.911474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:22.911393 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 03:28:22.911474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:22.911457 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 03:28:23.648914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.648879 5275 generic.go:356] "Generic (PLEG): container finished" podID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerID="26f156c0ed23e5940068c5e45401c871f99e58ac04fc7ed615c6aaf8f09ca2fa" exitCode=0 Nov 03 03:28:23.649336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.648956 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerDied","Data":"26f156c0ed23e5940068c5e45401c871f99e58ac04fc7ed615c6aaf8f09ca2fa"} Nov 03 03:28:23.649576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.649556 5275 scope.go:122] "RemoveContainer" containerID="26f156c0ed23e5940068c5e45401c871f99e58ac04fc7ed615c6aaf8f09ca2fa" Nov 03 03:28:23.651407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.651377 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:23.651495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.651455 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:23.651495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.651491 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:28:23.933271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.931358 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:23.933933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.933908 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:23.935021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:23.934734 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:28:24.374875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:24.368026 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:28:24.686952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:24.686827 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerStarted","Data":"edd4d58f11b8e9a92c0cba08742faca787ffaaf498bf03e46dd6368410255371"} Nov 03 03:28:25.001659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:25.001574 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:28:25.003019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:25.002994 5275 patch_prober.go:29] interesting pod/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 03:28:25.003141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:25.003044 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 03:28:26.001615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:26.001576 5275 patch_prober.go:29] interesting pod/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 03:28:26.002048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:26.001654 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 03:28:27.001742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:27.001704 5275 patch_prober.go:29] interesting pod/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 03:28:27.002163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:27.001769 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 03:28:28.001236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:28.001200 5275 patch_prober.go:29] interesting pod/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 03:28:28.001417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:28.001256 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 03:28:29.001190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:29.001151 5275 patch_prober.go:29] interesting pod/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 03:28:29.001597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:29.001206 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 03:28:30.051940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:30.051911 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:28:30.052422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:30.052401 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:28:30.053192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:30.053172 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:28:32.330510 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:28:32.330475 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:28:55.055000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:55.054915 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb"] Nov 03 03:28:55.103871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:55.103825 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c532295-tzbdb"] Nov 03 03:28:56.991558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:56.991519 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41102a61-589b-4bea-85fb-3cf5e6f64a4f" path="/var/lib/kubelet/pods/41102a61-589b-4bea-85fb-3cf5e6f64a4f/volumes" Nov 03 03:28:57.621007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.620968 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:28:57.627756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.627729 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.639697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.639159 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8lv\" (UniqueName: \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.639697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.639270 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.702820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.701160 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:28:57.791589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.791546 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8lv\" (UniqueName: \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.791755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.791652 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.792152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.792129 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-398727\"" Nov 03 03:28:57.792321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.792268 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-398727-dockercfg-kkjj9\"" Nov 03 03:28:57.819299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.819268 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.857890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.856630 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8lv\" (UniqueName: \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\") pod \"payload-processing-e2e-api-b-398727-78567d589c-kmqj7\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:57.947754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:57.947669 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:58.334388 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:28:58.334359 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07808c08_9603_428a_954b_3ab1b2d81801.slice/crio-3016c7b4402f770333ac4b292c7b52b56e9fefe74123f9919d5a1b856e09eea2 WatchSource:0}: Error finding container 3016c7b4402f770333ac4b292c7b52b56e9fefe74123f9919d5a1b856e09eea2: Status 404 returned error can't find the container with id 3016c7b4402f770333ac4b292c7b52b56e9fefe74123f9919d5a1b856e09eea2 Nov 03 03:28:58.334821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:58.334765 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:28:58.990441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:58.990402 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" event={"ID":"07808c08-9603-428a-954b-3ab1b2d81801","Type":"ContainerStarted","Data":"6cd9b9fe2bd19a9cafca739bca800f55961747c2739e6f6bd81cc18b6c1de7a1"} Nov 03 03:28:58.990664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:58.990462 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" event={"ID":"07808c08-9603-428a-954b-3ab1b2d81801","Type":"ContainerStarted","Data":"3016c7b4402f770333ac4b292c7b52b56e9fefe74123f9919d5a1b856e09eea2"} Nov 03 03:28:58.990664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:58.990535 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:28:59.110705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:28:59.110650 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" podStartSLOduration=2.110632748 podStartE2EDuration="2.110632748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:59.106150592 +0000 UTC m=+1630.258839012" watchObservedRunningTime="2026-11-03 03:28:59.110632748 +0000 UTC m=+1630.263321170" Nov 03 03:29:02.352056 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:29:02.352020 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:29:02.808015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.807973 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq"] Nov 03 03:29:02.812349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.812330 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.865551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.865513 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:29:02.872237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.872198 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.872425 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.872409 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5mg\" (UniqueName: \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.872564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.872542 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.914501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.914463 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:29:02.914694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.914503 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:29:02.914694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.914600 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:29:02.919784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.919763 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq"] Nov 03 03:29:02.973996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.973962 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5mg\" (UniqueName: \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.974183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.974079 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.974341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.974313 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.974907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.974877 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:02.976776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:02.976753 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:03.038492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:03.038455 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5mg\" (UniqueName: \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\") pod \"maas-api-revoke-sub-822a1b57-6zsnq\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:03.126814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:03.126715 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:03.515456 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:29:03.515420 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e094bab_ce26_45e5_abee_02f6efb4da51.slice/crio-37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef WatchSource:0}: Error finding container 37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef: Status 404 returned error can't find the container with id 37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef Nov 03 03:29:03.516922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:03.516892 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq"] Nov 03 03:29:04.037329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:04.037289 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" event={"ID":"0e094bab-ce26-45e5-abee-02f6efb4da51","Type":"ContainerStarted","Data":"c3e6a4197eac5b26ffda9066f1acfae89c74c53a84db9af3447b019d2589b1e9"} Nov 03 03:29:04.037522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:04.037355 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" event={"ID":"0e094bab-ce26-45e5-abee-02f6efb4da51","Type":"ContainerStarted","Data":"37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef"} Nov 03 03:29:04.156657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:04.156602 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" podStartSLOduration=2.156584904 podStartE2EDuration="2.156584904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:04.152064051 +0000 UTC m=+1635.304752472" watchObservedRunningTime="2026-11-03 03:29:04.156584904 +0000 UTC m=+1635.309273325" Nov 03 03:29:05.046952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:05.046917 5275 generic.go:356] "Generic (PLEG): container finished" podID="0e094bab-ce26-45e5-abee-02f6efb4da51" containerID="c3e6a4197eac5b26ffda9066f1acfae89c74c53a84db9af3447b019d2589b1e9" exitCode=0 Nov 03 03:29:05.047412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:05.047001 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" event={"ID":"0e094bab-ce26-45e5-abee-02f6efb4da51","Type":"ContainerDied","Data":"c3e6a4197eac5b26ffda9066f1acfae89c74c53a84db9af3447b019d2589b1e9"} Nov 03 03:29:06.335034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.335010 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:06.437240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.437205 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\") pod \"0e094bab-ce26-45e5-abee-02f6efb4da51\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " Nov 03 03:29:06.437435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.437329 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\" (UniqueName: \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\") pod \"0e094bab-ce26-45e5-abee-02f6efb4da51\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " Nov 03 03:29:06.437541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.437517 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca" pod "0e094bab-ce26-45e5-abee-02f6efb4da51" (UID: "0e094bab-ce26-45e5-abee-02f6efb4da51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:29:06.437602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.437547 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\") pod \"0e094bab-ce26-45e5-abee-02f6efb4da51\" (UID: \"0e094bab-ce26-45e5-abee-02f6efb4da51\") " Nov 03 03:29:06.438073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.438051 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:06.439987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.439956 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca" pod "0e094bab-ce26-45e5-abee-02f6efb4da51" (UID: "0e094bab-ce26-45e5-abee-02f6efb4da51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:29:06.440103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.440058 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg" pod "0e094bab-ce26-45e5-abee-02f6efb4da51" (UID: "0e094bab-ce26-45e5-abee-02f6efb4da51"). InnerVolumeSpecName "kube-api-access-gn5mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:29:06.539963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.539928 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e094bab-ce26-45e5-abee-02f6efb4da51-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:06.539963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:06.539962 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5mg\" (UniqueName: \"kubernetes.io/projected/0e094bab-ce26-45e5-abee-02f6efb4da51-kube-api-access-gn5mg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:07.066268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.066235 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" Nov 03 03:29:07.066466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.066239 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq" event={"ID":"0e094bab-ce26-45e5-abee-02f6efb4da51","Type":"ContainerDied","Data":"37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef"} Nov 03 03:29:07.066466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.066363 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37229a4229f28c12692e7876270aad0062fcc1392a5606639e02c2ad826163ef" Nov 03 03:29:07.900722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.900675 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:29:07.901393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.901377 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0e094bab-ce26-45e5-abee-02f6efb4da51" containerName="revoke-keys" Nov 03 03:29:07.901465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.901394 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e094bab-ce26-45e5-abee-02f6efb4da51" containerName="revoke-keys" Nov 03 03:29:07.901503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.901492 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e094bab-ce26-45e5-abee-02f6efb4da51" containerName="revoke-keys" Nov 03 03:29:07.905326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:07.905307 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.008055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.008020 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:29:08.058088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.058055 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59"] Nov 03 03:29:08.062287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.062259 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5e5d7-cnn59"] Nov 03 03:29:08.066041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.066310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066284 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.066457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066439 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.066584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066566 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.066674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066628 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.066920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066894 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.067012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.066937 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.067080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.067058 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.067202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.067185 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.168823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.168730 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.168823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.168778 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.168823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.168813 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.168904 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169033 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169283 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169344 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169523 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169559 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169354 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.169604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169598 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.170039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.169651 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.170418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.170393 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.170517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.170468 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.171599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.171570 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.172002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.171980 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.230236 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.230202 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.230396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.230338 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.519150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.519064 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:08.962705 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:29:08.962668 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfc344b_d698_49d7_b9c1_d0a0bf4aa49a.slice/crio-166e4e10b8c3e6f8b33f5056ad1652147efc76034e5984478cd90ee1321c5ea6 WatchSource:0}: Error finding container 166e4e10b8c3e6f8b33f5056ad1652147efc76034e5984478cd90ee1321c5ea6: Status 404 returned error can't find the container with id 166e4e10b8c3e6f8b33f5056ad1652147efc76034e5984478cd90ee1321c5ea6 Nov 03 03:29:08.963505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.963480 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:29:08.965543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.965506 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:08.965639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.965591 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:08.965639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.965620 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:08.989341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:08.989310 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fa4a6c-d363-438c-839c-3a155a5ae6a8" path="/var/lib/kubelet/pods/b7fa4a6c-d363-438c-839c-3a155a5ae6a8/volumes" Nov 03 03:29:09.091503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:09.091449 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" event={"ID":"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a","Type":"ContainerStarted","Data":"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2"} Nov 03 03:29:09.092664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:09.092637 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" event={"ID":"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a","Type":"ContainerStarted","Data":"166e4e10b8c3e6f8b33f5056ad1652147efc76034e5984478cd90ee1321c5ea6"} Nov 03 03:29:09.215077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:09.214977 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podStartSLOduration=2.214963312 podStartE2EDuration="2.214963312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:09.210349986 +0000 UTC m=+1640.363038408" watchObservedRunningTime="2026-11-03 03:29:09.214963312 +0000 UTC m=+1640.367651732" Nov 03 03:29:09.519324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:09.519246 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:10.001863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:10.001818 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:29:10.521883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:10.521833 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:10.522075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:10.521900 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:11.520173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:11.520140 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:11.520566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:11.520194 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:12.074982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:12.074946 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46"] Nov 03 03:29:12.099662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:12.099622 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a8da5-cmn46"] Nov 03 03:29:12.519802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:12.519719 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:12.519802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:12.519778 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:12.986590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:12.986554 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb5e3be-84fc-41e3-adec-0848199a73cb" path="/var/lib/kubelet/pods/bfb5e3be-84fc-41e3-adec-0848199a73cb/volumes" Nov 03 03:29:13.519539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:13.519505 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:13.519726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:13.519562 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:14.520520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:14.520376 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:14.520520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:14.520432 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:15.524913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:15.524870 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 03:29:15.525587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:15.525544 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 03:29:16.616452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:16.616417 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:16.617969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:16.617944 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:16.619565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:16.619545 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:29:19.051005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:19.050972 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74"] Nov 03 03:29:19.100327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:19.100295 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29efb7f0-kvw74"] Nov 03 03:29:20.991917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:20.991883 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a576e23-89de-440c-8335-1bbdaf924d23" path="/var/lib/kubelet/pods/7a576e23-89de-440c-8335-1bbdaf924d23/volumes" Nov 03 03:29:23.634684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.634649 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:29:23.641551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.641519 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:23.698983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.698943 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:29:23.749883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.741292 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-64c59d-dockercfg-mx8xk\"" Nov 03 03:29:23.749883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.741486 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-64c59d\"" Nov 03 03:29:23.771465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.771419 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9t4\" (UniqueName: \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:23.771908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.771864 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:23.894931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.873431 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9t4\" (UniqueName: \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:23.896247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.896210 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:23.905312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:23.905282 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:24.005881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:24.004203 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9t4\" (UniqueName: \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\") pod \"payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:24.260308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:24.260213 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:24.645752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:24.645720 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:29:24.646173 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:29:24.646015 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d55a1a_1c2a_48ab_b944_25e12c91a242.slice/crio-0597ae1a98935e4cb81ec9edf4a423330f4f7f9ddca7aabb5bdb9f2e1dca77de WatchSource:0}: Error finding container 0597ae1a98935e4cb81ec9edf4a423330f4f7f9ddca7aabb5bdb9f2e1dca77de: Status 404 returned error can't find the container with id 0597ae1a98935e4cb81ec9edf4a423330f4f7f9ddca7aabb5bdb9f2e1dca77de Nov 03 03:29:25.235555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.235490 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" event={"ID":"c8d55a1a-1c2a-48ab-b944-25e12c91a242","Type":"ContainerStarted","Data":"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990"} Nov 03 03:29:25.235945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.235863 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" event={"ID":"c8d55a1a-1c2a-48ab-b944-25e12c91a242","Type":"ContainerStarted","Data":"0597ae1a98935e4cb81ec9edf4a423330f4f7f9ddca7aabb5bdb9f2e1dca77de"} Nov 03 03:29:25.235945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.235917 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:25.353930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.353835 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" podStartSLOduration=2.353815553 podStartE2EDuration="2.353815553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:25.352634196 +0000 UTC m=+1656.505322622" watchObservedRunningTime="2026-11-03 03:29:25.353815553 +0000 UTC m=+1656.506503975" Nov 03 03:29:25.407240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.407207 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:29:25.411962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.411936 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.438808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.438772 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.438904 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.438957 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439094 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439219 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439264 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnm9v\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439290 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439325 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.439444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.439346 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.469200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.469168 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-372186-openshift-default-dockercfg-bk7lz\"" Nov 03 03:29:25.474045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.474013 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:29:25.547977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.547837 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.577024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.576980 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnm9v\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.577285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.577265 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.577456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.577441 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.577566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.577553 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.577686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.577668 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.599880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.577813 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.600202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.600175 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.600421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.600400 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.600673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.580188 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.600785 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.580768 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.600900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.578529 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.608894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.601313 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.612937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.578834 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.613131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.611652 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.633771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.614823 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.656872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.652595 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnm9v\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.685889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.680458 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\") pod \"e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:25.729544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:25.728159 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:26.088155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.088069 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8"] Nov 03 03:29:26.094733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.094709 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.146984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.146954 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:29:26.147255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.147183 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:29:26.147340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.147320 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:29:26.147533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.147515 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:29:26.151241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.151214 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8"] Nov 03 03:29:26.215508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.215465 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.215685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.215594 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.215794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.215769 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb9w\" (UniqueName: \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.314156 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:29:26.314125 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c76faf7_c4d1_4db5_9872_4ad2796ea655.slice/crio-6e1de8ef37ea4bc71b9d578eb5c79a52aa0d8d62c28f059f04847ddc2b5d343f WatchSource:0}: Error finding container 6e1de8ef37ea4bc71b9d578eb5c79a52aa0d8d62c28f059f04847ddc2b5d343f: Status 404 returned error can't find the container with id 6e1de8ef37ea4bc71b9d578eb5c79a52aa0d8d62c28f059f04847ddc2b5d343f Nov 03 03:29:26.314774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.314746 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:29:26.316771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.316751 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.316873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.316806 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.316944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.316914 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb9w\" (UniqueName: \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.317065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.317027 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:26.317127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.317110 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:26.317183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.317153 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:29:26.317507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.317488 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.319525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.319502 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.378950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.378923 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb9w\" (UniqueName: \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\") pod \"maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.409692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.409660 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:26.812417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:26.812386 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8"] Nov 03 03:29:26.813536 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:29:26.813513 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b226d79_fa00_4f40_9e53_bdd5244bed6d.slice/crio-555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9 WatchSource:0}: Error finding container 555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9: Status 404 returned error can't find the container with id 555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9 Nov 03 03:29:27.255458 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.255420 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerStarted","Data":"b09dcbe2ca9e9d74e622fbe500939a748695dc8b59245b0c86361079cd23b261"} Nov 03 03:29:27.255715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.255694 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerStarted","Data":"6e1de8ef37ea4bc71b9d578eb5c79a52aa0d8d62c28f059f04847ddc2b5d343f"} Nov 03 03:29:27.257534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.257509 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" event={"ID":"2b226d79-fa00-4f40-9e53-bdd5244bed6d","Type":"ContainerStarted","Data":"a1a358a89bde5e31d5f28727e7071aff790c535208eb3055f8a2b916fd621310"} Nov 03 03:29:27.257674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.257552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" event={"ID":"2b226d79-fa00-4f40-9e53-bdd5244bed6d","Type":"ContainerStarted","Data":"555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9"} Nov 03 03:29:27.383430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.383316 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podStartSLOduration=2.383302097 podStartE2EDuration="2.383302097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:27.379477897 +0000 UTC m=+1658.532166318" watchObservedRunningTime="2026-11-03 03:29:27.383302097 +0000 UTC m=+1658.535990517" Nov 03 03:29:27.494791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.494722 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" podStartSLOduration=1.494701989 podStartE2EDuration="1.494701989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:27.493966697 +0000 UTC m=+1658.646655131" watchObservedRunningTime="2026-11-03 03:29:27.494701989 +0000 UTC m=+1658.647390411" Nov 03 03:29:27.729368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.729277 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:27.734975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:27.734949 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:28.054206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.054174 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v"] Nov 03 03:29:28.058731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.058698 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ce11a0-xqd5v"] Nov 03 03:29:28.271629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.271598 5275 generic.go:356] "Generic (PLEG): container finished" podID="2b226d79-fa00-4f40-9e53-bdd5244bed6d" containerID="a1a358a89bde5e31d5f28727e7071aff790c535208eb3055f8a2b916fd621310" exitCode=0 Nov 03 03:29:28.271883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.271783 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" event={"ID":"2b226d79-fa00-4f40-9e53-bdd5244bed6d","Type":"ContainerDied","Data":"a1a358a89bde5e31d5f28727e7071aff790c535208eb3055f8a2b916fd621310"} Nov 03 03:29:28.272102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.272079 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:28.273319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.273302 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:29:28.990313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:28.990278 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60faca7e-f074-4b92-81d9-e6539c26f75c" path="/var/lib/kubelet/pods/60faca7e-f074-4b92-81d9-e6539c26f75c/volumes" Nov 03 03:29:29.465956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.465929 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:29.576978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.576874 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\") pod \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " Nov 03 03:29:29.577129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.577081 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\") pod \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " Nov 03 03:29:29.577178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.577130 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\" (UniqueName: \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\") pod \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\" (UID: \"2b226d79-fa00-4f40-9e53-bdd5244bed6d\") " Nov 03 03:29:29.577501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.577462 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca" pod "2b226d79-fa00-4f40-9e53-bdd5244bed6d" (UID: "2b226d79-fa00-4f40-9e53-bdd5244bed6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:29:29.578183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.578156 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:29.579311 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.579273 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca" pod "2b226d79-fa00-4f40-9e53-bdd5244bed6d" (UID: "2b226d79-fa00-4f40-9e53-bdd5244bed6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:29:29.579480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.579464 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w" pod "2b226d79-fa00-4f40-9e53-bdd5244bed6d" (UID: "2b226d79-fa00-4f40-9e53-bdd5244bed6d"). InnerVolumeSpecName "kube-api-access-whb9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:29:29.679826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.679784 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b226d79-fa00-4f40-9e53-bdd5244bed6d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:29.679826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:29.679827 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whb9w\" (UniqueName: \"kubernetes.io/projected/2b226d79-fa00-4f40-9e53-bdd5244bed6d-kube-api-access-whb9w\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:30.290259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:30.290227 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" Nov 03 03:29:30.290259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:30.290256 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8" event={"ID":"2b226d79-fa00-4f40-9e53-bdd5244bed6d","Type":"ContainerDied","Data":"555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9"} Nov 03 03:29:30.290538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:30.290301 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555081b201e7cde018eb523d3a8703d1363173e7875410b67edfbd4483d677b9" Nov 03 03:29:32.361252 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:29:32.361204 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:29:35.052130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:35.052094 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745"] Nov 03 03:29:35.101352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:35.101315 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d36f1a-mr745"] Nov 03 03:29:36.246712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:36.246684 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:29:36.989149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:36.989114 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378dd831-fa1d-480f-83aa-780d6e619eae" path="/var/lib/kubelet/pods/378dd831-fa1d-480f-83aa-780d6e619eae/volumes" Nov 03 03:29:37.346032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:37.345999 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:29:37.474001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:37.473966 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:29:42.346817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:42.346769 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" containerID="cri-o://e85c3d66900b75e99e1790b7c195c3aea057309b5fc44962d11351d711814256" gracePeriod=25 Nov 03 03:29:42.474703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:42.474634 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" containerID="cri-o://f092eafe27eed1bbc9e641948903ca3d3c45e6803a07161c82aa9514e262bd20" gracePeriod=25 Nov 03 03:29:45.078745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:45.078703 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7"] Nov 03 03:29:45.127787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:45.127753 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17506388-v5tw7"] Nov 03 03:29:45.361633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:45.361538 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:29:45.362569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:45.362532 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:29:46.986226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:46.986192 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7143caa-f22e-4e54-ba7e-0e5199a4d940" path="/var/lib/kubelet/pods/a7143caa-f22e-4e54-ba7e-0e5199a4d940/volumes" Nov 03 03:29:53.062067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:53.062036 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt"] Nov 03 03:29:53.111502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:53.111467 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19617288-zf2rt"] Nov 03 03:29:54.988980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:54.988946 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8842914-cf5e-4bee-8796-0d379b29a2d5" path="/var/lib/kubelet/pods/b8842914-cf5e-4bee-8796-0d379b29a2d5/volumes" Nov 03 03:29:55.361703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:55.361653 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:29:55.361944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:55.361731 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:29:57.051150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:57.051117 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm"] Nov 03 03:29:57.100956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:57.100924 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1bad90-sflsm"] Nov 03 03:29:58.988755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:29:58.988716 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e0fac2-3a39-43a7-8fc1-d57bf3060852" path="/var/lib/kubelet/pods/f6e0fac2-3a39-43a7-8fc1-d57bf3060852/volumes" Nov 03 03:30:00.174340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.174302 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:00.175410 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.175386 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2b226d79-fa00-4f40-9e53-bdd5244bed6d" containerName="revoke-keys" Nov 03 03:30:00.175463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.175416 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b226d79-fa00-4f40-9e53-bdd5244bed6d" containerName="revoke-keys" Nov 03 03:30:00.175659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.175643 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b226d79-fa00-4f40-9e53-bdd5244bed6d" containerName="revoke-keys" Nov 03 03:30:00.181953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.181921 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:00.182097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.182056 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.187480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.187457 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:00.187655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.187579 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.193482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.193461 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.224404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.224370 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:00.228805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.228782 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:00.228978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.228932 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.233923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.233891 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:00.234457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.234436 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.235620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.235568 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:30:00.239047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.239028 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.268790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.268763 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:30:00.268790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.268764 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:30:00.277150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.277120 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:00.285350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.285310 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:00.292749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.292710 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:00.308475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308444 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.308667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308488 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwvk\" (UniqueName: \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.308667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308512 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fkv\" (UniqueName: \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.308811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308688 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6z5\" (UniqueName: \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.308934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308802 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrpr\" (UniqueName: \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.308934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308876 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.309017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.308935 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.309149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.309127 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.309198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.309183 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.309379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.309354 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdx2\" (UniqueName: \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.309520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.309414 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.309520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.309488 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85rj\" (UniqueName: \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.329581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.329503 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:00.365286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.345770 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.542390 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543441 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdx2\" (UniqueName: \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543528 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543576 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85rj\" (UniqueName: \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543659 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543694 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwvk\" (UniqueName: \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543723 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fkv\" (UniqueName: \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543761 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6z5\" (UniqueName: \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543828 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrpr\" (UniqueName: \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543874 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.543914 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.544073 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.544119 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.545126 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.545162 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.545697 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.546866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.546303 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.558566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.558525 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.562212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.562144 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.606914 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrpr\" (UniqueName: \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\") pod \"maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.607126 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwvk\" (UniqueName: \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\") pod \"maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.610036 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6z5\" (UniqueName: \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\") pod \"maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.610036 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fkv\" (UniqueName: \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.612021 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85rj\" (UniqueName: \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.621929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.612682 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdx2\" (UniqueName: \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.800779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.800676 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:00.815291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.815066 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:00.817007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.816978 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:00.888997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.887175 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:00.888997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.887630 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:00.907516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:00.890010 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:01.555336 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.555305 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98c26c4_f056_4d81_a2fd_dea91250fb3b.slice/crio-de44ebcb49cd183cb887a6de83d249d1e262f863ec5217678cc3eeab2f290c1a WatchSource:0}: Error finding container de44ebcb49cd183cb887a6de83d249d1e262f863ec5217678cc3eeab2f290c1a: Status 404 returned error can't find the container with id de44ebcb49cd183cb887a6de83d249d1e262f863ec5217678cc3eeab2f290c1a Nov 03 03:30:01.555767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.555621 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:01.604763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.604736 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:01.604905 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.604885 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b5302e_495c_4aeb_abac_62618fc0ad90.slice/crio-db14a36940f30ff1607001b3ef4b72a510f384752bec4edd4d261e4e8a689b72 WatchSource:0}: Error finding container db14a36940f30ff1607001b3ef4b72a510f384752bec4edd4d261e4e8a689b72: Status 404 returned error can't find the container with id db14a36940f30ff1607001b3ef4b72a510f384752bec4edd4d261e4e8a689b72 Nov 03 03:30:01.606599 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.606565 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63fc88a_c9d4_4123_8069_97d51d63c464.slice/crio-4feff745e99518e16339080d6644aa57f31f923bfde45358d3616647a8649eba WatchSource:0}: Error finding container 4feff745e99518e16339080d6644aa57f31f923bfde45358d3616647a8649eba: Status 404 returned error can't find the container with id 4feff745e99518e16339080d6644aa57f31f923bfde45358d3616647a8649eba Nov 03 03:30:01.610549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.610529 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:01.617611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.617587 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:01.621964 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.621923 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492038a2_95be_49d1_887b_795b50cf1532.slice/crio-be813ed371e0f1d6ef8f66f3a85e7a88f83ba4dfe0b72a120558ebf188898cca WatchSource:0}: Error finding container be813ed371e0f1d6ef8f66f3a85e7a88f83ba4dfe0b72a120558ebf188898cca: Status 404 returned error can't find the container with id be813ed371e0f1d6ef8f66f3a85e7a88f83ba4dfe0b72a120558ebf188898cca Nov 03 03:30:01.657534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.657506 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:01.698025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.697997 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:01.708406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.708376 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerStarted","Data":"be813ed371e0f1d6ef8f66f3a85e7a88f83ba4dfe0b72a120558ebf188898cca"} Nov 03 03:30:01.709742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.709712 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerStarted","Data":"de44ebcb49cd183cb887a6de83d249d1e262f863ec5217678cc3eeab2f290c1a"} Nov 03 03:30:01.711057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.711034 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerStarted","Data":"4feff745e99518e16339080d6644aa57f31f923bfde45358d3616647a8649eba"} Nov 03 03:30:01.712287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:01.712257 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerStarted","Data":"db14a36940f30ff1607001b3ef4b72a510f384752bec4edd4d261e4e8a689b72"} Nov 03 03:30:01.752201 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.752167 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25a384d_9540_49de_8898_d7c582ea352c.slice/crio-14af43313dfd8b77043efa5467cf66433c2e83dac6af790b11c56931117de34e WatchSource:0}: Error finding container 14af43313dfd8b77043efa5467cf66433c2e83dac6af790b11c56931117de34e: Status 404 returned error can't find the container with id 14af43313dfd8b77043efa5467cf66433c2e83dac6af790b11c56931117de34e Nov 03 03:30:01.753088 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:01.752975 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3829269d_7b28_403e_b0bc_ff3cf8e7ce3d.slice/crio-e2d6ee4045dad540227f782812ff505dd9a255474e30816f8b9a44de7b5425d8 WatchSource:0}: Error finding container e2d6ee4045dad540227f782812ff505dd9a255474e30816f8b9a44de7b5425d8: Status 404 returned error can't find the container with id e2d6ee4045dad540227f782812ff505dd9a255474e30816f8b9a44de7b5425d8 Nov 03 03:30:02.383443 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:02.383407 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:30:02.731472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.731433 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerStarted","Data":"1c815331431add3aaf679d22d8eb0137822f03fb2d674266edffd7f67afd1a52"} Nov 03 03:30:02.733794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.733761 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerStarted","Data":"ae3dc013c86fddf4a285be295e577709b7cabf6a19dae5372cbd46e8bfd231b5"} Nov 03 03:30:02.736203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.736177 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerStarted","Data":"2f9cefacfe339d0f9a59626c1af193dc05a1bf815fde7e8de5464f1f65f56da5"} Nov 03 03:30:02.738429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.738293 5275 generic.go:356] "Generic (PLEG): container finished" podID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerID="f092eafe27eed1bbc9e641948903ca3d3c45e6803a07161c82aa9514e262bd20" exitCode=0 Nov 03 03:30:02.738429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.738367 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" event={"ID":"1fef473e-8c1d-467f-9831-c8946dfa0644","Type":"ContainerDied","Data":"f092eafe27eed1bbc9e641948903ca3d3c45e6803a07161c82aa9514e262bd20"} Nov 03 03:30:02.740185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.740136 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerStarted","Data":"46f8ba83a9ca7b143e609585d4501a86016ae0881a7c786421295cd831374608"} Nov 03 03:30:02.746004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.745968 5275 generic.go:356] "Generic (PLEG): container finished" podID="8b5b5ba1-1c46-4864-991c-24174b887081" containerID="e85c3d66900b75e99e1790b7c195c3aea057309b5fc44962d11351d711814256" exitCode=0 Nov 03 03:30:02.746067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.746032 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" event={"ID":"8b5b5ba1-1c46-4864-991c-24174b887081","Type":"ContainerDied","Data":"e85c3d66900b75e99e1790b7c195c3aea057309b5fc44962d11351d711814256"} Nov 03 03:30:02.748693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.747759 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerStarted","Data":"a6118667183ee3db49b31a13c083583d82dedbd3f3c4f9e56784e973dbaf6639"} Nov 03 03:30:02.748693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.747796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerStarted","Data":"e2d6ee4045dad540227f782812ff505dd9a255474e30816f8b9a44de7b5425d8"} Nov 03 03:30:02.751826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.751397 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerStarted","Data":"3c141f0d6f5ef304972c608b9dff90be9d848c578b0b39d28c9b0f904e29f65d"} Nov 03 03:30:02.751826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.751715 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerStarted","Data":"14af43313dfd8b77043efa5467cf66433c2e83dac6af790b11c56931117de34e"} Nov 03 03:30:02.877101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.872770 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" podStartSLOduration=2.872749689 podStartE2EDuration="2.872749689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:02.864251537 +0000 UTC m=+1694.016939960" watchObservedRunningTime="2026-11-03 03:30:02.872749689 +0000 UTC m=+1694.025438117" Nov 03 03:30:02.981901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:02.979964 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" podStartSLOduration=2.979944872 podStartE2EDuration="2.979944872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:02.979207918 +0000 UTC m=+1694.131896342" watchObservedRunningTime="2026-11-03 03:30:02.979944872 +0000 UTC m=+1694.132633291" Nov 03 03:30:03.003994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.003965 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:30:03.007721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.007695 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:30:03.059228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.059182 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7"] Nov 03 03:30:03.066577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.066546 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e008e559-4gxc7"] Nov 03 03:30:03.085899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.085861 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\" (UniqueName: \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\") pod \"8b5b5ba1-1c46-4864-991c-24174b887081\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " Nov 03 03:30:03.086061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.085934 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\") pod \"1fef473e-8c1d-467f-9831-c8946dfa0644\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " Nov 03 03:30:03.086148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086062 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\") pod \"8b5b5ba1-1c46-4864-991c-24174b887081\" (UID: \"8b5b5ba1-1c46-4864-991c-24174b887081\") " Nov 03 03:30:03.086202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086175 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\" (UniqueName: \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\") pod \"1fef473e-8c1d-467f-9831-c8946dfa0644\" (UID: \"1fef473e-8c1d-467f-9831-c8946dfa0644\") " Nov 03 03:30:03.086371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086346 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume" pod "1fef473e-8c1d-467f-9831-c8946dfa0644" (UID: "1fef473e-8c1d-467f-9831-c8946dfa0644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:03.086443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086394 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume" pod "8b5b5ba1-1c46-4864-991c-24174b887081" (UID: "8b5b5ba1-1c46-4864-991c-24174b887081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:03.086787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086766 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5b5ba1-1c46-4864-991c-24174b887081-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:03.086787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.086789 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fef473e-8c1d-467f-9831-c8946dfa0644-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:03.088170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.088151 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869" pod "8b5b5ba1-1c46-4864-991c-24174b887081" (UID: "8b5b5ba1-1c46-4864-991c-24174b887081"). InnerVolumeSpecName "kube-api-access-t8869". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:03.088546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.088527 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj" pod "1fef473e-8c1d-467f-9831-c8946dfa0644" (UID: "1fef473e-8c1d-467f-9831-c8946dfa0644"). InnerVolumeSpecName "kube-api-access-5vtlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:03.096784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.096734 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" podStartSLOduration=3.096721432 podStartE2EDuration="3.096721432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.093713126 +0000 UTC m=+1694.246401548" watchObservedRunningTime="2026-11-03 03:30:03.096721432 +0000 UTC m=+1694.249409893" Nov 03 03:30:03.188288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.188248 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtlj\" (UniqueName: \"kubernetes.io/projected/1fef473e-8c1d-467f-9831-c8946dfa0644-kube-api-access-5vtlj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:03.188288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.188289 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8869\" (UniqueName: \"kubernetes.io/projected/8b5b5ba1-1c46-4864-991c-24174b887081-kube-api-access-t8869\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:03.210878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.210809 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" podStartSLOduration=3.210792957 podStartE2EDuration="3.210792957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.20871342 +0000 UTC m=+1694.361401842" watchObservedRunningTime="2026-11-03 03:30:03.210792957 +0000 UTC m=+1694.363481377" Nov 03 03:30:03.330896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.330070 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" podStartSLOduration=3.330050726 podStartE2EDuration="3.330050726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.329820793 +0000 UTC m=+1694.482509214" watchObservedRunningTime="2026-11-03 03:30:03.330050726 +0000 UTC m=+1694.482739148" Nov 03 03:30:03.447093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.446994 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" podStartSLOduration=3.446979213 podStartE2EDuration="3.446979213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.444689033 +0000 UTC m=+1694.597377454" watchObservedRunningTime="2026-11-03 03:30:03.446979213 +0000 UTC m=+1694.599667634" Nov 03 03:30:03.765643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.765559 5275 generic.go:356] "Generic (PLEG): container finished" podID="b25a384d-9540-49de-8898-d7c582ea352c" containerID="3c141f0d6f5ef304972c608b9dff90be9d848c578b0b39d28c9b0f904e29f65d" exitCode=60 Nov 03 03:30:03.766156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.765642 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerDied","Data":"3c141f0d6f5ef304972c608b9dff90be9d848c578b0b39d28c9b0f904e29f65d"} Nov 03 03:30:03.766156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.766063 5275 scope.go:122] "RemoveContainer" containerID="3c141f0d6f5ef304972c608b9dff90be9d848c578b0b39d28c9b0f904e29f65d" Nov 03 03:30:03.767822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.767788 5275 generic.go:356] "Generic (PLEG): container finished" podID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerID="1c815331431add3aaf679d22d8eb0137822f03fb2d674266edffd7f67afd1a52" exitCode=60 Nov 03 03:30:03.767976 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.767937 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerDied","Data":"1c815331431add3aaf679d22d8eb0137822f03fb2d674266edffd7f67afd1a52"} Nov 03 03:30:03.768257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.768239 5275 scope.go:122] "RemoveContainer" containerID="1c815331431add3aaf679d22d8eb0137822f03fb2d674266edffd7f67afd1a52" Nov 03 03:30:03.770234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.770125 5275 generic.go:356] "Generic (PLEG): container finished" podID="492038a2-95be-49d1-887b-795b50cf1532" containerID="ae3dc013c86fddf4a285be295e577709b7cabf6a19dae5372cbd46e8bfd231b5" exitCode=60 Nov 03 03:30:03.770234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.770204 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerDied","Data":"ae3dc013c86fddf4a285be295e577709b7cabf6a19dae5372cbd46e8bfd231b5"} Nov 03 03:30:03.770428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.770413 5275 scope.go:122] "RemoveContainer" containerID="ae3dc013c86fddf4a285be295e577709b7cabf6a19dae5372cbd46e8bfd231b5" Nov 03 03:30:03.772281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.772261 5275 generic.go:356] "Generic (PLEG): container finished" podID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerID="2f9cefacfe339d0f9a59626c1af193dc05a1bf815fde7e8de5464f1f65f56da5" exitCode=60 Nov 03 03:30:03.772393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.772343 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerDied","Data":"2f9cefacfe339d0f9a59626c1af193dc05a1bf815fde7e8de5464f1f65f56da5"} Nov 03 03:30:03.772529 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.772520 5275 scope.go:122] "RemoveContainer" containerID="2f9cefacfe339d0f9a59626c1af193dc05a1bf815fde7e8de5464f1f65f56da5" Nov 03 03:30:03.774094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.774078 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" Nov 03 03:30:03.774191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.774116 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m" event={"ID":"1fef473e-8c1d-467f-9831-c8946dfa0644","Type":"ContainerDied","Data":"c8b2fde0c95ff1196ee17709ac804eddd5fa9f2cd19621d9e81176dfe454653d"} Nov 03 03:30:03.774191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.774174 5275 scope.go:122] "RemoveContainer" containerID="f092eafe27eed1bbc9e641948903ca3d3c45e6803a07161c82aa9514e262bd20" Nov 03 03:30:03.796016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.795987 5275 generic.go:356] "Generic (PLEG): container finished" podID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerID="46f8ba83a9ca7b143e609585d4501a86016ae0881a7c786421295cd831374608" exitCode=60 Nov 03 03:30:03.797249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.796308 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerDied","Data":"46f8ba83a9ca7b143e609585d4501a86016ae0881a7c786421295cd831374608"} Nov 03 03:30:03.797249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.796556 5275 scope.go:122] "RemoveContainer" containerID="46f8ba83a9ca7b143e609585d4501a86016ae0881a7c786421295cd831374608" Nov 03 03:30:03.800438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.800417 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" event={"ID":"8b5b5ba1-1c46-4864-991c-24174b887081","Type":"ContainerDied","Data":"ae31f9aef57fbf998cbf934d1ce2d3faf1aebad531e3a46d03d2949a654013a8"} Nov 03 03:30:03.800546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.800525 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl" Nov 03 03:30:03.803966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.803943 5275 generic.go:356] "Generic (PLEG): container finished" podID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerID="a6118667183ee3db49b31a13c083583d82dedbd3f3c4f9e56784e973dbaf6639" exitCode=60 Nov 03 03:30:03.804075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.803978 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerDied","Data":"a6118667183ee3db49b31a13c083583d82dedbd3f3c4f9e56784e973dbaf6639"} Nov 03 03:30:03.804382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.804360 5275 scope.go:122] "RemoveContainer" containerID="a6118667183ee3db49b31a13c083583d82dedbd3f3c4f9e56784e973dbaf6639" Nov 03 03:30:03.855427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.855404 5275 scope.go:122] "RemoveContainer" containerID="e85c3d66900b75e99e1790b7c195c3aea057309b5fc44962d11351d711814256" Nov 03 03:30:03.996410 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:03.996376 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:30:04.002310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.002281 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-dhfrl"] Nov 03 03:30:04.478589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.478536 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:30:04.485408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.485376 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-w722m"] Nov 03 03:30:04.815465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.815433 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerStarted","Data":"c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28"} Nov 03 03:30:04.818597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.818572 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerStarted","Data":"104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef"} Nov 03 03:30:04.821609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.821584 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerStarted","Data":"8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed"} Nov 03 03:30:04.823435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.823411 5275 generic.go:356] "Generic (PLEG): container finished" podID="b25a384d-9540-49de-8898-d7c582ea352c" containerID="f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3" exitCode=60 Nov 03 03:30:04.823538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.823486 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerDied","Data":"f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3"} Nov 03 03:30:04.823538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.823532 5275 scope.go:122] "RemoveContainer" containerID="3c141f0d6f5ef304972c608b9dff90be9d848c578b0b39d28c9b0f904e29f65d" Nov 03 03:30:04.823733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.823719 5275 scope.go:122] "RemoveContainer" containerID="f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3" Nov 03 03:30:04.824050 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:04.824029 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr_odh-ai-gateway-infra(b25a384d-9540-49de-8898-d7c582ea352c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" podUID="b25a384d-9540-49de-8898-d7c582ea352c" Nov 03 03:30:04.825482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.825461 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerStarted","Data":"f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f"} Nov 03 03:30:04.827297 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.827278 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerStarted","Data":"5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c"} Nov 03 03:30:04.988969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.988930 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" path="/var/lib/kubelet/pods/1fef473e-8c1d-467f-9831-c8946dfa0644/volumes" Nov 03 03:30:04.989742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.989712 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2" path="/var/lib/kubelet/pods/5ea8aca3-9a0a-49b3-ba03-d1c6d35bc4a2/volumes" Nov 03 03:30:04.990383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:04.990360 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" path="/var/lib/kubelet/pods/8b5b5ba1-1c46-4864-991c-24174b887081/volumes" Nov 03 03:30:05.838061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.838029 5275 generic.go:356] "Generic (PLEG): container finished" podID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerID="104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef" exitCode=60 Nov 03 03:30:05.838554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.838122 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerDied","Data":"104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef"} Nov 03 03:30:05.838554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.838175 5275 scope.go:122] "RemoveContainer" containerID="46f8ba83a9ca7b143e609585d4501a86016ae0881a7c786421295cd831374608" Nov 03 03:30:05.838554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.838502 5275 scope.go:122] "RemoveContainer" containerID="104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef" Nov 03 03:30:05.838960 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.838929 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv_odh-ai-gateway-infra(e63fc88a-c9d4-4123-8069-97d51d63c464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" Nov 03 03:30:05.840385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.840365 5275 generic.go:356] "Generic (PLEG): container finished" podID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerID="8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed" exitCode=60 Nov 03 03:30:05.840506 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.840446 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerDied","Data":"8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed"} Nov 03 03:30:05.840739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.840722 5275 scope.go:122] "RemoveContainer" containerID="8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed" Nov 03 03:30:05.841120 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.841097 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22_odh-ai-gateway-infra(3829269d-7b28-403e-b0bc-ff3cf8e7ce3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" Nov 03 03:30:05.842655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.842626 5275 scope.go:122] "RemoveContainer" containerID="f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3" Nov 03 03:30:05.842961 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.842939 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr_odh-ai-gateway-infra(b25a384d-9540-49de-8898-d7c582ea352c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" podUID="b25a384d-9540-49de-8898-d7c582ea352c" Nov 03 03:30:05.845575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.845551 5275 generic.go:356] "Generic (PLEG): container finished" podID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerID="f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f" exitCode=60 Nov 03 03:30:05.845688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.845648 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerDied","Data":"f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f"} Nov 03 03:30:05.845897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.845882 5275 scope.go:122] "RemoveContainer" containerID="f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f" Nov 03 03:30:05.846144 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.846122 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj_odh-ai-gateway-infra(65b5302e-495c-4aeb-abac-62618fc0ad90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" Nov 03 03:30:05.847909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.847891 5275 generic.go:356] "Generic (PLEG): container finished" podID="492038a2-95be-49d1-887b-795b50cf1532" containerID="5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c" exitCode=60 Nov 03 03:30:05.848002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.847969 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerDied","Data":"5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c"} Nov 03 03:30:05.848227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.848207 5275 scope.go:122] "RemoveContainer" containerID="5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c" Nov 03 03:30:05.848550 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.848523 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf_odh-ai-gateway-infra(492038a2-95be-49d1-887b-795b50cf1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" podUID="492038a2-95be-49d1-887b-795b50cf1532" Nov 03 03:30:05.850095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.850073 5275 generic.go:356] "Generic (PLEG): container finished" podID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerID="c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28" exitCode=60 Nov 03 03:30:05.850187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.850157 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerDied","Data":"c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28"} Nov 03 03:30:05.850329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.850315 5275 scope.go:122] "RemoveContainer" containerID="c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28" Nov 03 03:30:05.850598 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:05.850578 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l_odh-ai-gateway-infra(a98c26c4-f056-4d81-a2fd-dea91250fb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" Nov 03 03:30:05.854491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.854475 5275 scope.go:122] "RemoveContainer" containerID="a6118667183ee3db49b31a13c083583d82dedbd3f3c4f9e56784e973dbaf6639" Nov 03 03:30:05.873814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.873786 5275 scope.go:122] "RemoveContainer" containerID="1c815331431add3aaf679d22d8eb0137822f03fb2d674266edffd7f67afd1a52" Nov 03 03:30:05.889347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.889318 5275 scope.go:122] "RemoveContainer" containerID="ae3dc013c86fddf4a285be295e577709b7cabf6a19dae5372cbd46e8bfd231b5" Nov 03 03:30:05.903975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:05.903953 5275 scope.go:122] "RemoveContainer" containerID="2f9cefacfe339d0f9a59626c1af193dc05a1bf815fde7e8de5464f1f65f56da5" Nov 03 03:30:06.862139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:06.862106 5275 scope.go:122] "RemoveContainer" containerID="c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28" Nov 03 03:30:06.863344 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:06.862533 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l_odh-ai-gateway-infra(a98c26c4-f056-4d81-a2fd-dea91250fb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" Nov 03 03:30:06.865049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:06.865024 5275 scope.go:122] "RemoveContainer" containerID="104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef" Nov 03 03:30:06.865459 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:06.865432 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv_odh-ai-gateway-infra(e63fc88a-c9d4-4123-8069-97d51d63c464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" Nov 03 03:30:06.867704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:06.867684 5275 scope.go:122] "RemoveContainer" containerID="8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed" Nov 03 03:30:06.868119 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:06.868092 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22_odh-ai-gateway-infra(3829269d-7b28-403e-b0bc-ff3cf8e7ce3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" Nov 03 03:30:06.869776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:06.869749 5275 scope.go:122] "RemoveContainer" containerID="f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f" Nov 03 03:30:06.870143 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:06.870107 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj_odh-ai-gateway-infra(65b5302e-495c-4aeb-abac-62618fc0ad90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" Nov 03 03:30:06.871582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:06.871563 5275 scope.go:122] "RemoveContainer" containerID="5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c" Nov 03 03:30:06.872039 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:06.871902 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf_odh-ai-gateway-infra(492038a2-95be-49d1-887b-795b50cf1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" podUID="492038a2-95be-49d1-887b-795b50cf1532" Nov 03 03:30:07.982261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.982225 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n"] Nov 03 03:30:07.982946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.982930 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" Nov 03 03:30:07.983010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.982946 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" Nov 03 03:30:07.983010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.982984 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" Nov 03 03:30:07.983010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.982990 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" Nov 03 03:30:07.983118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.983108 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fef473e-8c1d-467f-9831-c8946dfa0644" containerName="payload-pre-processing" Nov 03 03:30:07.983149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.983126 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5b5ba1-1c46-4864-991c-24174b887081" containerName="payload-processing" Nov 03 03:30:07.988170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:07.988147 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.041213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.041179 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:30:08.045578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.045545 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n"] Nov 03 03:30:08.069894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.069840 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.070064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.070018 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8stw\" (UniqueName: \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.070157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.070127 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.172299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.172261 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.172484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.172450 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8stw\" (UniqueName: \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.172585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.172567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.173402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.173368 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.174885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.174865 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.233725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.233652 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8stw\" (UniqueName: \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\") pod \"maas-api-revoke-sub-805d7ae6-l7c7n\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.304320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.304283 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:08.705023 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:08.704991 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a914b79_2c67_48ca_8150_98d0030c326d.slice/crio-1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b WatchSource:0}: Error finding container 1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b: Status 404 returned error can't find the container with id 1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b Nov 03 03:30:08.706061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.705956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n"] Nov 03 03:30:08.904804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:08.904767 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" event={"ID":"1a914b79-2c67-48ca-8150-98d0030c326d","Type":"ContainerStarted","Data":"1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b"} Nov 03 03:30:09.915199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:09.915109 5275 generic.go:356] "Generic (PLEG): container finished" podID="1a914b79-2c67-48ca-8150-98d0030c326d" containerID="d121ad33175c567ee19464df6979c8e64f361c5dc2872867f89de509d94681b1" exitCode=0 Nov 03 03:30:09.915662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:09.915190 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" event={"ID":"1a914b79-2c67-48ca-8150-98d0030c326d","Type":"ContainerDied","Data":"d121ad33175c567ee19464df6979c8e64f361c5dc2872867f89de509d94681b1"} Nov 03 03:30:11.154747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.154718 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:11.240876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.240751 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\") pod \"1a914b79-2c67-48ca-8150-98d0030c326d\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " Nov 03 03:30:11.241042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.240929 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\" (UniqueName: \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\") pod \"1a914b79-2c67-48ca-8150-98d0030c326d\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " Nov 03 03:30:11.241042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.241001 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\") pod \"1a914b79-2c67-48ca-8150-98d0030c326d\" (UID: \"1a914b79-2c67-48ca-8150-98d0030c326d\") " Nov 03 03:30:11.241498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.241366 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca" pod "1a914b79-2c67-48ca-8150-98d0030c326d" (UID: "1a914b79-2c67-48ca-8150-98d0030c326d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:11.241731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.241694 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:11.243684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.243643 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca" pod "1a914b79-2c67-48ca-8150-98d0030c326d" (UID: "1a914b79-2c67-48ca-8150-98d0030c326d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:11.246286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.246265 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw" pod "1a914b79-2c67-48ca-8150-98d0030c326d" (UID: "1a914b79-2c67-48ca-8150-98d0030c326d"). InnerVolumeSpecName "kube-api-access-h8stw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:11.342986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.342955 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a914b79-2c67-48ca-8150-98d0030c326d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:11.342986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.342984 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8stw\" (UniqueName: \"kubernetes.io/projected/1a914b79-2c67-48ca-8150-98d0030c326d-kube-api-access-h8stw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:11.944972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.944941 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" Nov 03 03:30:11.945129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.944978 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n" event={"ID":"1a914b79-2c67-48ca-8150-98d0030c326d","Type":"ContainerDied","Data":"1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b"} Nov 03 03:30:11.945129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:11.945051 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe969d3a331ad0c2e0ffca2f1d211257756df804cb08bb54410540c125fba1b" Nov 03 03:30:14.065527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:14.065407 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh"] Nov 03 03:30:14.100392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:14.100326 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa7e47a-q2jjh"] Nov 03 03:30:14.986875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:14.986828 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88318b6-e3d0-4df0-98bd-eb4624471977" path="/var/lib/kubelet/pods/c88318b6-e3d0-4df0-98bd-eb4624471977/volumes" Nov 03 03:30:15.978624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:15.978585 5275 scope.go:122] "RemoveContainer" containerID="f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3" Nov 03 03:30:15.979070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:15.978721 5275 scope.go:122] "RemoveContainer" containerID="c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28" Nov 03 03:30:15.979070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:15.979000 5275 scope.go:122] "RemoveContainer" containerID="5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c" Nov 03 03:30:17.053114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.053065 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerStarted","Data":"5ec2f750ee56528f601f37a572608b994205458611f3aa5f5e9051d70038b316"} Nov 03 03:30:17.055715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.055684 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerStarted","Data":"52476a4a5dc19f465fa584aec34c3b7c66d1493cbba89195f8e2655a88a7bacc"} Nov 03 03:30:17.058168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.058144 5275 generic.go:356] "Generic (PLEG): container finished" podID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerID="d9d08bd9d0d39272e85137611b4a8882f1975e2944231277d00108330dfd7962" exitCode=60 Nov 03 03:30:17.058281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.058208 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerDied","Data":"d9d08bd9d0d39272e85137611b4a8882f1975e2944231277d00108330dfd7962"} Nov 03 03:30:17.058325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.058278 5275 scope.go:122] "RemoveContainer" containerID="c629e434866d7a885c5f03514852dd62bb41e98bcd8276a8a01d27a2892c8f28" Nov 03 03:30:17.058666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:17.058652 5275 scope.go:122] "RemoveContainer" containerID="d9d08bd9d0d39272e85137611b4a8882f1975e2944231277d00108330dfd7962" Nov 03 03:30:17.059009 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:17.058992 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l_odh-ai-gateway-infra(a98c26c4-f056-4d81-a2fd-dea91250fb3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" Nov 03 03:30:18.068793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.068760 5275 generic.go:356] "Generic (PLEG): container finished" podID="b25a384d-9540-49de-8898-d7c582ea352c" containerID="5ec2f750ee56528f601f37a572608b994205458611f3aa5f5e9051d70038b316" exitCode=60 Nov 03 03:30:18.069317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.068835 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerDied","Data":"5ec2f750ee56528f601f37a572608b994205458611f3aa5f5e9051d70038b316"} Nov 03 03:30:18.069317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.068911 5275 scope.go:122] "RemoveContainer" containerID="f8a8daef5036278e1aa6f179a15c701d2310c0d21bf4e1ddf3b13db9ede36cc3" Nov 03 03:30:18.069478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.069328 5275 scope.go:122] "RemoveContainer" containerID="5ec2f750ee56528f601f37a572608b994205458611f3aa5f5e9051d70038b316" Nov 03 03:30:18.069729 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:18.069705 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr_odh-ai-gateway-infra(b25a384d-9540-49de-8898-d7c582ea352c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" podUID="b25a384d-9540-49de-8898-d7c582ea352c" Nov 03 03:30:18.071108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.071089 5275 generic.go:356] "Generic (PLEG): container finished" podID="492038a2-95be-49d1-887b-795b50cf1532" containerID="52476a4a5dc19f465fa584aec34c3b7c66d1493cbba89195f8e2655a88a7bacc" exitCode=60 Nov 03 03:30:18.071186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.071120 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerDied","Data":"52476a4a5dc19f465fa584aec34c3b7c66d1493cbba89195f8e2655a88a7bacc"} Nov 03 03:30:18.071539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.071521 5275 scope.go:122] "RemoveContainer" containerID="52476a4a5dc19f465fa584aec34c3b7c66d1493cbba89195f8e2655a88a7bacc" Nov 03 03:30:18.071907 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:18.071886 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf_odh-ai-gateway-infra(492038a2-95be-49d1-887b-795b50cf1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" podUID="492038a2-95be-49d1-887b-795b50cf1532" Nov 03 03:30:18.082805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.082784 5275 scope.go:122] "RemoveContainer" containerID="5f47b75c2c4c751c555ca6f830fc83e6bb3232d65019576dc4261fbe2c56d32c" Nov 03 03:30:18.185134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.185102 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:18.320342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.320256 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:18.435999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.435966 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:18.665370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.665341 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:18.797739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.797699 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\" (UniqueName: \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\") pod \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " Nov 03 03:30:18.797929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.797814 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\") pod \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\" (UID: \"a98c26c4-f056-4d81-a2fd-dea91250fb3b\") " Nov 03 03:30:18.798360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.798331 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca" pod "a98c26c4-f056-4d81-a2fd-dea91250fb3b" (UID: "a98c26c4-f056-4d81-a2fd-dea91250fb3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:18.798797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.798774 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98c26c4-f056-4d81-a2fd-dea91250fb3b-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:18.800256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.800226 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv" pod "a98c26c4-f056-4d81-a2fd-dea91250fb3b" (UID: "a98c26c4-f056-4d81-a2fd-dea91250fb3b"). InnerVolumeSpecName "kube-api-access-z5fkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:18.900542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.900457 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5fkv\" (UniqueName: \"kubernetes.io/projected/a98c26c4-f056-4d81-a2fd-dea91250fb3b-kube-api-access-z5fkv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:18.982825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.982794 5275 scope.go:122] "RemoveContainer" containerID="8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed" Nov 03 03:30:18.983096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.983075 5275 scope.go:122] "RemoveContainer" containerID="f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f" Nov 03 03:30:18.983229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:18.983191 5275 scope.go:122] "RemoveContainer" containerID="104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef" Nov 03 03:30:19.094211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.093901 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" event={"ID":"a98c26c4-f056-4d81-a2fd-dea91250fb3b","Type":"ContainerDied","Data":"de44ebcb49cd183cb887a6de83d249d1e262f863ec5217678cc3eeab2f290c1a"} Nov 03 03:30:19.094211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.093954 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l" Nov 03 03:30:19.094211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.094147 5275 scope.go:122] "RemoveContainer" containerID="d9d08bd9d0d39272e85137611b4a8882f1975e2944231277d00108330dfd7962" Nov 03 03:30:19.294371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.294342 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:19.381910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.381876 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:19.389952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.389917 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894610-l7k5l"] Nov 03 03:30:19.408220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.408189 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\" (UniqueName: \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\") pod \"b25a384d-9540-49de-8898-d7c582ea352c\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " Nov 03 03:30:19.408431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.408283 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\") pod \"b25a384d-9540-49de-8898-d7c582ea352c\" (UID: \"b25a384d-9540-49de-8898-d7c582ea352c\") " Nov 03 03:30:19.408660 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.408635 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca" pod "b25a384d-9540-49de-8898-d7c582ea352c" (UID: "b25a384d-9540-49de-8898-d7c582ea352c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:19.408908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.408890 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25a384d-9540-49de-8898-d7c582ea352c-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:19.450095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.449988 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2" pod "b25a384d-9540-49de-8898-d7c582ea352c" (UID: "b25a384d-9540-49de-8898-d7c582ea352c"). InnerVolumeSpecName "kube-api-access-mjdx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:19.463967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.463946 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:19.509492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.509456 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\") pod \"492038a2-95be-49d1-887b-795b50cf1532\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " Nov 03 03:30:19.509677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.509598 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\" (UniqueName: \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\") pod \"492038a2-95be-49d1-887b-795b50cf1532\" (UID: \"492038a2-95be-49d1-887b-795b50cf1532\") " Nov 03 03:30:19.509907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.509874 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca" pod "492038a2-95be-49d1-887b-795b50cf1532" (UID: "492038a2-95be-49d1-887b-795b50cf1532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:19.510409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.510391 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/492038a2-95be-49d1-887b-795b50cf1532-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:19.510458 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.510414 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjdx2\" (UniqueName: \"kubernetes.io/projected/b25a384d-9540-49de-8898-d7c582ea352c-kube-api-access-mjdx2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:19.511939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.511920 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5" pod "492038a2-95be-49d1-887b-795b50cf1532" (UID: "492038a2-95be-49d1-887b-795b50cf1532"). InnerVolumeSpecName "kube-api-access-jf6z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:19.611689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:19.611661 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6z5\" (UniqueName: \"kubernetes.io/projected/492038a2-95be-49d1-887b-795b50cf1532-kube-api-access-jf6z5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:20.120005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.119966 5275 generic.go:356] "Generic (PLEG): container finished" podID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerID="a15cf4bf863a3eaa75271ed47062e6bad06ec56bd5d82f49f5afef1ca4a67a75" exitCode=60 Nov 03 03:30:20.120479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.120083 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerDied","Data":"a15cf4bf863a3eaa75271ed47062e6bad06ec56bd5d82f49f5afef1ca4a67a75"} Nov 03 03:30:20.120479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.120146 5275 scope.go:122] "RemoveContainer" containerID="104e9759c82a8c913e6b9b19123a384fa491c51e55eff288a0b21a755b0fd3ef" Nov 03 03:30:20.120611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.120483 5275 scope.go:122] "RemoveContainer" containerID="a15cf4bf863a3eaa75271ed47062e6bad06ec56bd5d82f49f5afef1ca4a67a75" Nov 03 03:30:20.120746 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:20.120718 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv_odh-ai-gateway-infra(e63fc88a-c9d4-4123-8069-97d51d63c464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" Nov 03 03:30:20.123072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.123046 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerStarted","Data":"b64e0e2952e0f890df04456c29cf494b7fac060c1aeab2dd8e8e4732126e5d90"} Nov 03 03:30:20.124605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.124574 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" Nov 03 03:30:20.124605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.124586 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr" event={"ID":"b25a384d-9540-49de-8898-d7c582ea352c","Type":"ContainerDied","Data":"14af43313dfd8b77043efa5467cf66433c2e83dac6af790b11c56931117de34e"} Nov 03 03:30:20.126906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.126881 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerStarted","Data":"5f304219b648846435e958b7f01260182193a137bef98d0e3943ae824732ae88"} Nov 03 03:30:20.128477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.128460 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" Nov 03 03:30:20.128578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.128458 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf" event={"ID":"492038a2-95be-49d1-887b-795b50cf1532","Type":"ContainerDied","Data":"be813ed371e0f1d6ef8f66f3a85e7a88f83ba4dfe0b72a120558ebf188898cca"} Nov 03 03:30:20.140104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.140083 5275 scope.go:122] "RemoveContainer" containerID="5ec2f750ee56528f601f37a572608b994205458611f3aa5f5e9051d70038b316" Nov 03 03:30:20.151327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.151302 5275 scope.go:122] "RemoveContainer" containerID="52476a4a5dc19f465fa584aec34c3b7c66d1493cbba89195f8e2655a88a7bacc" Nov 03 03:30:20.467792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.467706 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:20.473326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.473294 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b562f5-29894610-f5cxf"] Nov 03 03:30:20.697465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.697425 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:20.701707 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.701681 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894610-9fvlr"] Nov 03 03:30:20.988255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.988223 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492038a2-95be-49d1-887b-795b50cf1532" path="/var/lib/kubelet/pods/492038a2-95be-49d1-887b-795b50cf1532/volumes" Nov 03 03:30:20.988712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.988698 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" path="/var/lib/kubelet/pods/a98c26c4-f056-4d81-a2fd-dea91250fb3b/volumes" Nov 03 03:30:20.989151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:20.989136 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25a384d-9540-49de-8898-d7c582ea352c" path="/var/lib/kubelet/pods/b25a384d-9540-49de-8898-d7c582ea352c/volumes" Nov 03 03:30:21.140810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.140777 5275 generic.go:356] "Generic (PLEG): container finished" podID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerID="b64e0e2952e0f890df04456c29cf494b7fac060c1aeab2dd8e8e4732126e5d90" exitCode=60 Nov 03 03:30:21.141306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.140867 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerDied","Data":"b64e0e2952e0f890df04456c29cf494b7fac060c1aeab2dd8e8e4732126e5d90"} Nov 03 03:30:21.141306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.140926 5275 scope.go:122] "RemoveContainer" containerID="8c60ab6065b0f7be8b0716501002a35f32f6cd86ce34e37770745db683b518ed" Nov 03 03:30:21.141427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.141383 5275 scope.go:122] "RemoveContainer" containerID="b64e0e2952e0f890df04456c29cf494b7fac060c1aeab2dd8e8e4732126e5d90" Nov 03 03:30:21.141770 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:21.141743 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22_odh-ai-gateway-infra(3829269d-7b28-403e-b0bc-ff3cf8e7ce3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" Nov 03 03:30:21.144091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.144068 5275 generic.go:356] "Generic (PLEG): container finished" podID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerID="5f304219b648846435e958b7f01260182193a137bef98d0e3943ae824732ae88" exitCode=60 Nov 03 03:30:21.144176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.144140 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerDied","Data":"5f304219b648846435e958b7f01260182193a137bef98d0e3943ae824732ae88"} Nov 03 03:30:21.144529 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.144515 5275 scope.go:122] "RemoveContainer" containerID="5f304219b648846435e958b7f01260182193a137bef98d0e3943ae824732ae88" Nov 03 03:30:21.145075 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:21.144872 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj_odh-ai-gateway-infra(65b5302e-495c-4aeb-abac-62618fc0ad90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" Nov 03 03:30:21.156376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.156355 5275 scope.go:122] "RemoveContainer" containerID="f9e18d3586eed321dc62701430479c9c20d86e7d7ad493c77972457d403be12f" Nov 03 03:30:21.249586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.249504 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:21.368913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.366362 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:21.431455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.431429 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:21.540436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.540400 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\" (UniqueName: \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\") pod \"e63fc88a-c9d4-4123-8069-97d51d63c464\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " Nov 03 03:30:21.540719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.540698 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\") pod \"e63fc88a-c9d4-4123-8069-97d51d63c464\" (UID: \"e63fc88a-c9d4-4123-8069-97d51d63c464\") " Nov 03 03:30:21.541071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.541042 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca" pod "e63fc88a-c9d4-4123-8069-97d51d63c464" (UID: "e63fc88a-c9d4-4123-8069-97d51d63c464"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:21.541321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.541309 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc88a-c9d4-4123-8069-97d51d63c464-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:21.542731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.542709 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj" pod "e63fc88a-c9d4-4123-8069-97d51d63c464" (UID: "e63fc88a-c9d4-4123-8069-97d51d63c464"). InnerVolumeSpecName "kube-api-access-p85rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:21.596173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.596137 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:21.643058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:21.643017 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p85rj\" (UniqueName: \"kubernetes.io/projected/e63fc88a-c9d4-4123-8069-97d51d63c464-kube-api-access-p85rj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:22.155705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.155674 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" Nov 03 03:30:22.156185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.155676 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv" event={"ID":"e63fc88a-c9d4-4123-8069-97d51d63c464","Type":"ContainerDied","Data":"4feff745e99518e16339080d6644aa57f31f923bfde45358d3616647a8649eba"} Nov 03 03:30:22.156185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.155799 5275 scope.go:122] "RemoveContainer" containerID="a15cf4bf863a3eaa75271ed47062e6bad06ec56bd5d82f49f5afef1ca4a67a75" Nov 03 03:30:22.361064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.360982 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:22.370882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.370835 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:22.457967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.457543 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\") pod \"65b5302e-495c-4aeb-abac-62618fc0ad90\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " Nov 03 03:30:22.457967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.457769 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\") pod \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " Nov 03 03:30:22.458183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.458137 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\" (UniqueName: \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\") pod \"65b5302e-495c-4aeb-abac-62618fc0ad90\" (UID: \"65b5302e-495c-4aeb-abac-62618fc0ad90\") " Nov 03 03:30:22.458323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.458304 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\" (UniqueName: \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\") pod \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\" (UID: \"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d\") " Nov 03 03:30:22.458778 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.458751 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca" pod "65b5302e-495c-4aeb-abac-62618fc0ad90" (UID: "65b5302e-495c-4aeb-abac-62618fc0ad90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:22.458949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.458921 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca" pod "3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" (UID: "3829269d-7b28-403e-b0bc-ff3cf8e7ce3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:22.459137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.459103 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:22.459137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.459129 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b5302e-495c-4aeb-abac-62618fc0ad90-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:22.461108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.461083 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr" pod "3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" (UID: "3829269d-7b28-403e-b0bc-ff3cf8e7ce3d"). InnerVolumeSpecName "kube-api-access-fkrpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:22.461534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.461517 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk" pod "65b5302e-495c-4aeb-abac-62618fc0ad90" (UID: "65b5302e-495c-4aeb-abac-62618fc0ad90"). InnerVolumeSpecName "kube-api-access-2lwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:22.561186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.561147 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwvk\" (UniqueName: \"kubernetes.io/projected/65b5302e-495c-4aeb-abac-62618fc0ad90-kube-api-access-2lwvk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:22.561186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.561186 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrpr\" (UniqueName: \"kubernetes.io/projected/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d-kube-api-access-fkrpr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:22.583466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.583426 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:22.594799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.594768 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894610-55wvv"] Nov 03 03:30:22.990287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:22.990247 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" path="/var/lib/kubelet/pods/e63fc88a-c9d4-4123-8069-97d51d63c464/volumes" Nov 03 03:30:23.055191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.055157 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2"] Nov 03 03:30:23.061982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.061947 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08c166b-7mlb2"] Nov 03 03:30:23.175362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.175337 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" Nov 03 03:30:23.175362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.175342 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj" event={"ID":"65b5302e-495c-4aeb-abac-62618fc0ad90","Type":"ContainerDied","Data":"db14a36940f30ff1607001b3ef4b72a510f384752bec4edd4d261e4e8a689b72"} Nov 03 03:30:23.175979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.175411 5275 scope.go:122] "RemoveContainer" containerID="5f304219b648846435e958b7f01260182193a137bef98d0e3943ae824732ae88" Nov 03 03:30:23.178897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.178835 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" event={"ID":"3829269d-7b28-403e-b0bc-ff3cf8e7ce3d","Type":"ContainerDied","Data":"e2d6ee4045dad540227f782812ff505dd9a255474e30816f8b9a44de7b5425d8"} Nov 03 03:30:23.179024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.178911 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22" Nov 03 03:30:23.190426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.190400 5275 scope.go:122] "RemoveContainer" containerID="b64e0e2952e0f890df04456c29cf494b7fac060c1aeab2dd8e8e4732126e5d90" Nov 03 03:30:23.341930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.341892 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:23.347350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.347316 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-398727-29894610-x8jsj"] Nov 03 03:30:23.456074 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.456043 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:23.462723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:23.462696 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d82a79-29894610-nwz22"] Nov 03 03:30:24.987283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:24.987251 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" path="/var/lib/kubelet/pods/3829269d-7b28-403e-b0bc-ff3cf8e7ce3d/volumes" Nov 03 03:30:24.987790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:24.987775 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" path="/var/lib/kubelet/pods/65b5302e-495c-4aeb-abac-62618fc0ad90/volumes" Nov 03 03:30:24.988314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:24.988296 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a6b169-6fbb-4517-94fb-dd10405f103e" path="/var/lib/kubelet/pods/a5a6b169-6fbb-4517-94fb-dd10405f103e/volumes" Nov 03 03:30:25.052545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:25.052510 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp"] Nov 03 03:30:25.101094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:25.101064 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7224be-q55zp"] Nov 03 03:30:26.986883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:26.986835 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c311e4e-2879-4c34-a671-225732c55c28" path="/var/lib/kubelet/pods/6c311e4e-2879-4c34-a671-225732c55c28/volumes" Nov 03 03:30:32.332527 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:30:32.332471 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:30:33.052046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:33.052007 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc"] Nov 03 03:30:33.100730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:33.100695 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1bc38f-mp6zc"] Nov 03 03:30:34.985739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:34.985702 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474530e2-6abf-43ee-8be2-c0dd4ddec402" path="/var/lib/kubelet/pods/474530e2-6abf-43ee-8be2-c0dd4ddec402/volumes" Nov 03 03:30:35.053219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:35.053188 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd"] Nov 03 03:30:35.102129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:35.102097 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb37775-dv9vd"] Nov 03 03:30:36.987521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:36.987481 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677722d4-2b90-4d36-b10c-8115a99c16e2" path="/var/lib/kubelet/pods/677722d4-2b90-4d36-b10c-8115a99c16e2/volumes" Nov 03 03:30:45.054137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:45.054105 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m"] Nov 03 03:30:45.104669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:45.104618 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-375aa143-hkf9m"] Nov 03 03:30:46.989241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:46.989178 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee72008-e392-48b3-aed9-53de33ebf731" path="/var/lib/kubelet/pods/4ee72008-e392-48b3-aed9-53de33ebf731/volumes" Nov 03 03:30:49.054649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.054613 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26"] Nov 03 03:30:49.104498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.104459 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c71f4a70-dqq26"] Nov 03 03:30:49.691483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.691450 5275 scope.go:122] "RemoveContainer" containerID="34c58cd6ae37a857ca87c9fbc4bfd06d9e4e94a410d982dff53e1f968af6d426" Nov 03 03:30:49.703274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.703244 5275 scope.go:122] "RemoveContainer" containerID="b06468cf1469de7aded9755825f40a2e6f706ea9cd208ad004d6b31b2a490ae7" Nov 03 03:30:49.713637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.713611 5275 scope.go:122] "RemoveContainer" containerID="1ab578d79c9b94fb945ab741109804dd98f7ed0ae934db2e0d4165ec7260120a" Nov 03 03:30:49.723985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.723963 5275 scope.go:122] "RemoveContainer" containerID="2a9089fc209ee0e972cea63233732369e8fbe275defb9c70a48bd716400586a6" Nov 03 03:30:49.734791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.734768 5275 scope.go:122] "RemoveContainer" containerID="a8381f6f6026cfea278d07ceb56c98056057a535213e0ceda9cca7da734aa841" Nov 03 03:30:49.746100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.745973 5275 scope.go:122] "RemoveContainer" containerID="004ee378ee55a01f9f44f7781e2da2ab57e9eba156b7b2e6837aae91b417aaa1" Nov 03 03:30:49.761531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.761502 5275 scope.go:122] "RemoveContainer" containerID="987353185d775e7f47dfafb5c16038540ba8ff3277b23a4509d1a6ade87d6161" Nov 03 03:30:49.775019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:49.774994 5275 scope.go:122] "RemoveContainer" containerID="5a0a282998725c65dcdfb479e686ef26520b8acee8f3ae9ec90b27d62ab6114d" Nov 03 03:30:50.615324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615296 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:30:50.615954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615931 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.615954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615947 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615958 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615963 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615972 5275 cpu_manager.go:414] "containerMap: removing container" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615978 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615989 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.615993 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616000 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616004 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616012 5275 cpu_manager.go:414] "containerMap: removing container" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616016 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616025 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616029 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616038 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616043 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616051 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616056 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616064 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616069 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616077 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1a914b79-2c67-48ca-8150-98d0030c326d" containerName="revoke-keys" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616081 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a914b79-2c67-48ca-8150-98d0030c326d" containerName="revoke-keys" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616094 5275 cpu_manager.go:414] "containerMap: removing container" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616098 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616104 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616109 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616121 5275 cpu_manager.go:414] "containerMap: removing container" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616125 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616133 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616137 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616233 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616242 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616247 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616252 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616256 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616261 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a914b79-2c67-48ca-8150-98d0030c326d" containerName="revoke-keys" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616267 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616273 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616280 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616284 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616290 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63fc88a-c9d4-4123-8069-97d51d63c464" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616296 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616302 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616307 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616312 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:30:50.616510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.616318 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:30:50.622009 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.621988 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.674903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.674838 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-398727-openshift-default-dockercfg-ckr6k\"" Nov 03 03:30:50.723690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.723646 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.723893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.723872 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.723911 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.723998 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.724059 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.724106 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.724129 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.724179 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.724238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.724203 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.730598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.730569 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:30:50.825403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825367 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825409 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825466 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825508 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825540 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825562 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825612 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.825723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825627 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.825744 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.826062 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.826109 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.826156 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.826318 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.826403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.826311 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.828801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.828772 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.829050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.829018 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.886683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.886594 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.887216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.887193 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.936487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.936445 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:50.988650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:50.988613 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b449cf6-1294-4b9b-9974-7553d623590a" path="/var/lib/kubelet/pods/0b449cf6-1294-4b9b-9974-7553d623590a/volumes" Nov 03 03:30:51.379092 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:30:51.379058 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014fae9a_881d_4dc3_8209_45baf1b31029.slice/crio-6c1dad355a0c4100e106b737d5772165631efc1a91d41b001cc85bf27486f6ad WatchSource:0}: Error finding container 6c1dad355a0c4100e106b737d5772165631efc1a91d41b001cc85bf27486f6ad: Status 404 returned error can't find the container with id 6c1dad355a0c4100e106b737d5772165631efc1a91d41b001cc85bf27486f6ad Nov 03 03:30:51.383494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.383453 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:30:51.383606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.383529 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:30:51.383606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.383558 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:30:51.387967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.387939 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:30:51.466540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.466504 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" event={"ID":"014fae9a-881d-4dc3-8209-45baf1b31029","Type":"ContainerStarted","Data":"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277"} Nov 03 03:30:51.466776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.466753 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" event={"ID":"014fae9a-881d-4dc3-8209-45baf1b31029","Type":"ContainerStarted","Data":"6c1dad355a0c4100e106b737d5772165631efc1a91d41b001cc85bf27486f6ad"} Nov 03 03:30:51.586557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.586490 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podStartSLOduration=1.586470161 podStartE2EDuration="1.586470161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:51.585550326 +0000 UTC m=+1742.738238747" watchObservedRunningTime="2026-11-03 03:30:51.586470161 +0000 UTC m=+1742.739158583" Nov 03 03:30:51.937175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:51.937139 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:52.938304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:52.938263 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:52.939220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:52.938322 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:53.937362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:53.937323 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:53.937550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:53.937389 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:54.937821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:54.937789 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:54.938232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:54.937931 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:55.939499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:55.936873 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:55.939499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:55.936955 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:56.944382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:56.944339 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:56.976979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:56.944401 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:57.937459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:57.937417 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 03:30:57.937652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:57.937490 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 03:30:58.942403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:58.942370 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:58.942831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:58.942784 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:30:58.943398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:30:58.943380 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:31:02.335052 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:31:02.334989 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:31:07.959636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:07.959599 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:31:07.960237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:07.959899 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" containerID="cri-o://8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a" gracePeriod=30 Nov 03 03:31:09.050486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.050451 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8"] Nov 03 03:31:09.099923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.099894 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902a0c2-xwxz8"] Nov 03 03:31:09.913021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.912984 5275 generic.go:356] "Generic (PLEG): container finished" podID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerID="b09dcbe2ca9e9d74e622fbe500939a748695dc8b59245b0c86361079cd23b261" exitCode=0 Nov 03 03:31:09.913219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.913066 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerDied","Data":"b09dcbe2ca9e9d74e622fbe500939a748695dc8b59245b0c86361079cd23b261"} Nov 03 03:31:09.913687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.913666 5275 scope.go:122] "RemoveContainer" containerID="b09dcbe2ca9e9d74e622fbe500939a748695dc8b59245b0c86361079cd23b261" Nov 03 03:31:09.915478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.915444 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:31:09.915583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.915522 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:31:09.915583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:09.915556 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:31:10.064228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:10.064161 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht"] Nov 03 03:31:10.076326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:10.076291 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a53ca6a-98xht"] Nov 03 03:31:10.931293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:10.931254 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerStarted","Data":"156b4eb2d5c22f2cab0ddb489722192d501a90ebfe6678f86dd609af4a7509c5"} Nov 03 03:31:10.986611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:10.986578 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b226d79-fa00-4f40-9e53-bdd5244bed6d" path="/var/lib/kubelet/pods/2b226d79-fa00-4f40-9e53-bdd5244bed6d/volumes" Nov 03 03:31:10.987183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:10.987166 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3419e1-7143-4259-a4d2-6aec5866035a" path="/var/lib/kubelet/pods/6f3419e1-7143-4259-a4d2-6aec5866035a/volumes" Nov 03 03:31:11.729328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:11.729284 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:31:11.730715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:11.730691 5275 patch_prober.go:29] interesting pod/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 03:31:11.730838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:11.730742 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 03:31:12.729376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:12.729340 5275 patch_prober.go:29] interesting pod/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 03:31:12.729790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:12.729399 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 03:31:13.288418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.285967 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:31:13.457484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.457061 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:31:13.519281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519249 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519298 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519327 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519467 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519614 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519706 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519667 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519704 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519771 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.519918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.519825 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\") pod \"3e7dd91e-595f-4629-86e0-803084960b13\" (UID: \"3e7dd91e-595f-4629-86e0-803084960b13\") " Nov 03 03:31:13.520721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.520231 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.520721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.520486 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.520721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.520678 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.520721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.520617 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:13.521063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.520955 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.521256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.521231 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.523011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.522976 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:13.523312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.523291 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.524203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.524178 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:13.524312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.524215 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k" pod "3e7dd91e-595f-4629-86e0-803084960b13" (UID: "3e7dd91e-595f-4629-86e0-803084960b13"). InnerVolumeSpecName "kube-api-access-lkt6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:13.621939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621836 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.621939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621895 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7dd91e-595f-4629-86e0-803084960b13-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.621939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621911 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkt6k\" (UniqueName: \"kubernetes.io/projected/3e7dd91e-595f-4629-86e0-803084960b13-kube-api-access-lkt6k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.621939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621925 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.621939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621939 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.622212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621954 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.622212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621966 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7dd91e-595f-4629-86e0-803084960b13-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.622212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.621979 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7dd91e-595f-4629-86e0-803084960b13-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.729695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.729657 5275 patch_prober.go:29] interesting pod/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 03:31:13.730198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:13.729759 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 03:31:14.005734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.005631 5275 generic.go:356] "Generic (PLEG): container finished" podID="3e7dd91e-595f-4629-86e0-803084960b13" containerID="8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a" exitCode=0 Nov 03 03:31:14.005915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.005778 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" event={"ID":"3e7dd91e-595f-4629-86e0-803084960b13","Type":"ContainerDied","Data":"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a"} Nov 03 03:31:14.005915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.005832 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" event={"ID":"3e7dd91e-595f-4629-86e0-803084960b13","Type":"ContainerDied","Data":"ce8c1802e746474a682b55783f7fe8db5b9ae0a57525311ee692206a143fd417"} Nov 03 03:31:14.005915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.005885 5275 scope.go:122] "RemoveContainer" containerID="8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a" Nov 03 03:31:14.006119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.006059 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x" Nov 03 03:31:14.023583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.023247 5275 scope.go:122] "RemoveContainer" containerID="8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a" Nov 03 03:31:14.027898 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:31:14.027867 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a\": container with ID starting with 8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a not found: ID does not exist" containerID="8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a" Nov 03 03:31:14.028002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.027913 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a"} err="failed to get container status \"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a\": rpc error: code = NotFound desc = could not find container \"8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a\": container with ID starting with 8fd52dfcd49382d13f120b0dab533971d2b6efd58c0c4cfc4795fdd8af13d28a not found: ID does not exist" Nov 03 03:31:14.237674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.237631 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:31:14.289953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.272933 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-j457x"] Nov 03 03:31:14.729006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.728927 5275 patch_prober.go:29] interesting pod/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 03:31:14.729006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.728991 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 03:31:14.993077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:14.992990 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7dd91e-595f-4629-86e0-803084960b13" path="/var/lib/kubelet/pods/3e7dd91e-595f-4629-86e0-803084960b13/volumes" Nov 03 03:31:15.084602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:15.084416 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd"] Nov 03 03:31:15.091217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:15.091184 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25dca97-crlkd"] Nov 03 03:31:15.729366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:15.729331 5275 patch_prober.go:29] interesting pod/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 03:31:15.729558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:15.729401 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 03:31:16.734507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:16.734470 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:31:16.735597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:16.735574 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:31:16.988417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:16.988330 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2" path="/var/lib/kubelet/pods/8d2e7a3e-7f69-448c-ab1f-f555ec6c83c2/volumes" Nov 03 03:31:26.059871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:26.057660 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8"] Nov 03 03:31:26.106648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:26.106600 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de8f982-92wr8"] Nov 03 03:31:26.987715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:26.987677 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fd9546-bdd6-4e59-a030-2115031e8e25" path="/var/lib/kubelet/pods/37fd9546-bdd6-4e59-a030-2115031e8e25/volumes" Nov 03 03:31:32.336113 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:31:32.336076 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:31:35.132909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.132870 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8"] Nov 03 03:31:35.134468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134437 5275 cpu_manager.go:414] "containerMap: removing container" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:31:35.134669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134654 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:31:35.134884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134800 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:31:35.134884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134821 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25a384d-9540-49de-8898-d7c582ea352c" containerName="cleanup" Nov 03 03:31:35.134884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134836 5275 cpu_manager.go:414] "containerMap: removing container" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:31:35.134884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134862 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:31:35.134884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134896 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134904 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134934 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.134946 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3829269d-7b28-403e-b0bc-ff3cf8e7ce3d" containerName="cleanup" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.135118 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="492038a2-95be-49d1-887b-795b50cf1532" containerName="cleanup" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.135143 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b5302e-495c-4aeb-abac-62618fc0ad90" containerName="cleanup" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.135154 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98c26c4-f056-4d81-a2fd-dea91250fb3b" containerName="cleanup" Nov 03 03:31:35.135247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.135161 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7dd91e-595f-4629-86e0-803084960b13" containerName="istio-proxy" Nov 03 03:31:35.141996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.141965 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.193109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.193078 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8"] Nov 03 03:31:35.235610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.235579 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:31:35.235796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.235629 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:31:35.254105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.254071 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkfs\" (UniqueName: \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.254280 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.254188 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.254337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.254277 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.276474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.276442 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:31:35.285193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.285173 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:31:35.355354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.355317 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.355527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.355401 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.355610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.355596 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkfs\" (UniqueName: \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.356312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.356218 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.358037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.358015 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.422510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.422438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkfs\" (UniqueName: \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\") pod \"maas-api-revoke-sub-259d06d8-pw4g8\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.465596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.465559 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:35.853340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:35.853300 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8"] Nov 03 03:31:35.854406 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:31:35.854379 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf98ed0b_d5b4_413e_b0ea_c7d846ee14e7.slice/crio-35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728 WatchSource:0}: Error finding container 35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728: Status 404 returned error can't find the container with id 35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728 Nov 03 03:31:36.313428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:36.313383 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" event={"ID":"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7","Type":"ContainerStarted","Data":"4131397ef451deb3e0efcaf858d3435f150058e607fd97b824f7d2dbdecf0b68"} Nov 03 03:31:36.313922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:36.313450 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" event={"ID":"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7","Type":"ContainerStarted","Data":"35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728"} Nov 03 03:31:36.446920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:36.446752 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" podStartSLOduration=1.446734817 podStartE2EDuration="1.446734817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:36.445330435 +0000 UTC m=+1787.598018857" watchObservedRunningTime="2026-11-03 03:31:36.446734817 +0000 UTC m=+1787.599423238" Nov 03 03:31:37.328299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:37.328078 5275 generic.go:356] "Generic (PLEG): container finished" podID="df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" containerID="4131397ef451deb3e0efcaf858d3435f150058e607fd97b824f7d2dbdecf0b68" exitCode=0 Nov 03 03:31:37.328299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:37.328128 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" event={"ID":"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7","Type":"ContainerDied","Data":"4131397ef451deb3e0efcaf858d3435f150058e607fd97b824f7d2dbdecf0b68"} Nov 03 03:31:38.187818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.187772 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:31:38.188158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.188121 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" containerID="cri-o://edd4d58f11b8e9a92c0cba08742faca787ffaaf498bf03e46dd6368410255371" gracePeriod=30 Nov 03 03:31:38.662860 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.662815 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:38.683142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.683111 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\") pod \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " Nov 03 03:31:38.683331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.683229 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\" (UniqueName: \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\") pod \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " Nov 03 03:31:38.683473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.683453 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\") pod \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\" (UID: \"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7\") " Nov 03 03:31:38.683535 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.683475 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca" pod "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" (UID: "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:38.684041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.684020 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:38.685498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.685477 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs" pod "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" (UID: "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7"). InnerVolumeSpecName "kube-api-access-pwkfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:38.685692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.685669 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca" pod "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" (UID: "df98ed0b-d5b4-413e-b0ea-c7d846ee14e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:38.785205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.785175 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:38.785205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:38.785203 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkfs\" (UniqueName: \"kubernetes.io/projected/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7-kube-api-access-pwkfs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:39.346914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:39.346880 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" event={"ID":"df98ed0b-d5b4-413e-b0ea-c7d846ee14e7","Type":"ContainerDied","Data":"35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728"} Nov 03 03:31:39.347130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:39.346939 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35795d26293d09cf5d7ca5b969005dd348feae7c0564cc7b230f78a3a5325728" Nov 03 03:31:39.347130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:39.347005 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8" Nov 03 03:31:43.451332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.451292 5275 generic.go:356] "Generic (PLEG): container finished" podID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerID="edd4d58f11b8e9a92c0cba08742faca787ffaaf498bf03e46dd6368410255371" exitCode=0 Nov 03 03:31:43.451793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.451519 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerDied","Data":"edd4d58f11b8e9a92c0cba08742faca787ffaaf498bf03e46dd6368410255371"} Nov 03 03:31:43.451793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.451580 5275 scope.go:122] "RemoveContainer" containerID="26f156c0ed23e5940068c5e45401c871f99e58ac04fc7ed615c6aaf8f09ca2fa" Nov 03 03:31:43.513832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.513804 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:31:43.557896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.557857 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.557953 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.557987 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558135 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558206 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558249 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:43.558325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558260 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558482 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:43.558777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558532 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558570 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558607 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\") pod \"8b7aa980-d3fc-4024-b511-642e6f7ff719\" (UID: \"8b7aa980-d3fc-4024-b511-642e6f7ff719\") " Nov 03 03:31:43.558777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558607 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:43.559094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.558842 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:43.559094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559040 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:43.559603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559582 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.559725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559606 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.559725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559622 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.559725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559636 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.559725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.559650 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7aa980-d3fc-4024-b511-642e6f7ff719-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.561070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.561035 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "kube-api-access-v5n6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:43.561474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.561451 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:43.561474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.561468 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:43.561617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.561450 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token" pod "8b7aa980-d3fc-4024-b511-642e6f7ff719" (UID: "8b7aa980-d3fc-4024-b511-642e6f7ff719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:43.661555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.661516 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.661555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.661550 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5n6b\" (UniqueName: \"kubernetes.io/projected/8b7aa980-d3fc-4024-b511-642e6f7ff719-kube-api-access-v5n6b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.661555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.661561 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:43.661802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:43.661570 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7aa980-d3fc-4024-b511-642e6f7ff719-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:44.462013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.461985 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" Nov 03 03:31:44.462013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.462002 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9" event={"ID":"8b7aa980-d3fc-4024-b511-642e6f7ff719","Type":"ContainerDied","Data":"c5b7d29bc30cf5c8a0dfac2d93e8cc45eede71f00abf4301463a0b63d9973c22"} Nov 03 03:31:44.462581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.462071 5275 scope.go:122] "RemoveContainer" containerID="edd4d58f11b8e9a92c0cba08742faca787ffaaf498bf03e46dd6368410255371" Nov 03 03:31:44.678755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.678657 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:31:44.728224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.728187 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b562f5-openshift-default-7bd8fb77b5-jvgz9"] Nov 03 03:31:44.988037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:44.987953 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" path="/var/lib/kubelet/pods/8b7aa980-d3fc-4024-b511-642e6f7ff719/volumes" Nov 03 03:31:45.144490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.144453 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:31:45.144790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.144763 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" containerID="cri-o://de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a" gracePeriod=30 Nov 03 03:31:45.148321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.148295 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:31:45.148560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.148539 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" containerID="cri-o://3991e9dc521e164e8929d45018879740ac98672d517335b9010e1b7e47b8e93c" gracePeriod=30 Nov 03 03:31:45.195077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.195040 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:31:45.195327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.195301 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" containerID="cri-o://1c0d9984c8435d90364602fdb8b4642d30f2d05b07a404e454e6bc353525fb4e" gracePeriod=30 Nov 03 03:31:45.201695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.201665 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:31:45.201995 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:45.201966 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" containerID="cri-o://25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b" gracePeriod=30 Nov 03 03:31:46.063405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:46.063370 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:31:48.878837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:48.878791 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:31:48.879262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:48.878905 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:31:49.500708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.500677 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:31:49.503120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.503098 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:31:49.506769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.506747 5275 kubelet.go:1689] "Image garbage collection succeeded" Nov 03 03:31:49.921323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.921284 5275 scope.go:122] "RemoveContainer" containerID="f50ef4faea0fa5e69b23370e20a1fa5683c22f8b428bf288c376d66b1f684c34" Nov 03 03:31:49.974252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.974142 5275 scope.go:122] "RemoveContainer" containerID="28e394adfda1a0ef8b1a63247ad4ac6a8888b473e27c7fbddc79ee430d50a015" Nov 03 03:31:49.994191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:49.994166 5275 scope.go:122] "RemoveContainer" containerID="2b9767fe00a77d179f12dd774e9cd44704a007e3738f0e02dcb82d9e0697d609" Nov 03 03:31:50.007717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.007690 5275 scope.go:122] "RemoveContainer" containerID="a7612005487f8fad36a6e58afba01ecb053f9426374a283df17a0cf87134a7df" Nov 03 03:31:50.021905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.021876 5275 scope.go:122] "RemoveContainer" containerID="793a3e44b95d02ff7ff514a4aa1ba21c5252d741e168a391e88c5e2e4cea38b8" Nov 03 03:31:50.037559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.037530 5275 scope.go:122] "RemoveContainer" containerID="5cfa8482c2c294805e3050b34e7077eb3eb8e21e21c4edf2a40d9c178d6b303c" Nov 03 03:31:50.049062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.049026 5275 scope.go:122] "RemoveContainer" containerID="fd94be92959801348124fcb50857607465271b254a38cf7e89e757a1fe28605d" Nov 03 03:31:50.062781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.062753 5275 scope.go:122] "RemoveContainer" containerID="de23698ad29d810c1b74090509559fcd11dc43dd8c994aba688be5d007dc915b" Nov 03 03:31:50.075079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.075051 5275 scope.go:122] "RemoveContainer" containerID="298a4c1c8e5721d8bbbf586f8d87f9e33217594432a49faf0bf8ce7a89d9ca4f" Nov 03 03:31:50.227644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.227602 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 03:31:50.227832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.227686 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 03:31:50.578078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.577077 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:31:50.643991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.643905 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:31:50.646659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646622 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646695 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646736 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646778 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646813 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646922 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.646962 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647023 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647125 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647162 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647216 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647251 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647289 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647393 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647441 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\") pod \"a07bf80a-6769-4df6-9e65-bca09a30152a\" (UID: \"a07bf80a-6769-4df6-9e65-bca09a30152a\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647491 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647532 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.647637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.647616 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\") pod \"7d850884-8836-4708-8b5a-b6fe51ee418d\" (UID: \"7d850884-8836-4708-8b5a-b6fe51ee418d\") " Nov 03 03:31:50.648698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.648669 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.651093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.648913 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.651093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.649125 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.651093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.649700 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:50.651093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.651037 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.651910 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.652660 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.656955 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.662186 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.663357 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "kube-api-access-czv8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.663614 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.666892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.664157 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.675626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.675572 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.677094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.677043 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.690863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.690566 5275 generic.go:356] "Generic (PLEG): container finished" podID="80512c30-b976-4e03-93d3-f106775a5d85" containerID="3991e9dc521e164e8929d45018879740ac98672d517335b9010e1b7e47b8e93c" exitCode=0 Nov 03 03:31:50.690863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.690748 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" event={"ID":"80512c30-b976-4e03-93d3-f106775a5d85","Type":"ContainerDied","Data":"3991e9dc521e164e8929d45018879740ac98672d517335b9010e1b7e47b8e93c"} Nov 03 03:31:50.693906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.693380 5275 generic.go:356] "Generic (PLEG): container finished" podID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerID="de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a" exitCode=0 Nov 03 03:31:50.693906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.693520 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" Nov 03 03:31:50.693906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.693681 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" event={"ID":"a07bf80a-6769-4df6-9e65-bca09a30152a","Type":"ContainerDied","Data":"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a"} Nov 03 03:31:50.693906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.693725 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27" event={"ID":"a07bf80a-6769-4df6-9e65-bca09a30152a","Type":"ContainerDied","Data":"7ee6f8e2a1e778e31cb744de7e641e481d3df708bd8024db47f3efbff3b359f5"} Nov 03 03:31:50.693906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.693754 5275 scope.go:122] "RemoveContainer" containerID="de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a" Nov 03 03:31:50.698486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.697769 5275 generic.go:356] "Generic (PLEG): container finished" podID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerID="25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b" exitCode=0 Nov 03 03:31:50.698486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.698033 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" event={"ID":"7d850884-8836-4708-8b5a-b6fe51ee418d","Type":"ContainerDied","Data":"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b"} Nov 03 03:31:50.698486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.698082 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" event={"ID":"7d850884-8836-4708-8b5a-b6fe51ee418d","Type":"ContainerDied","Data":"c4b7892a5c2b2f957517613e86d6fcd28b782d6d7acf225a064e68ad29c4dfb8"} Nov 03 03:31:50.698486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.698190 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf" Nov 03 03:31:50.699612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.699573 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.699725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.699600 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:50.700688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.700655 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo" pod "a07bf80a-6769-4df6-9e65-bca09a30152a" (UID: "a07bf80a-6769-4df6-9e65-bca09a30152a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:50.702041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.701977 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st" pod "7d850884-8836-4708-8b5a-b6fe51ee418d" (UID: "7d850884-8836-4708-8b5a-b6fe51ee418d"). InnerVolumeSpecName "kube-api-access-g57st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.707869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.706380 5275 generic.go:356] "Generic (PLEG): container finished" podID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerID="1c0d9984c8435d90364602fdb8b4642d30f2d05b07a404e454e6bc353525fb4e" exitCode=0 Nov 03 03:31:50.707869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.706424 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" event={"ID":"f9fd9bba-181d-4e48-bb2d-1585b2c39258","Type":"ContainerDied","Data":"1c0d9984c8435d90364602fdb8b4642d30f2d05b07a404e454e6bc353525fb4e"} Nov 03 03:31:50.712834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.712656 5275 scope.go:122] "RemoveContainer" containerID="de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a" Nov 03 03:31:50.713624 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:31:50.713354 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a\": container with ID starting with de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a not found: ID does not exist" containerID="de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a" Nov 03 03:31:50.713624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.713389 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a"} err="failed to get container status \"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a\": rpc error: code = NotFound desc = could not find container \"de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a\": container with ID starting with de43ec80cd4dcda4431b8a7cff5a8208b83695b8f6eab5cb6a20405d0e34402a not found: ID does not exist" Nov 03 03:31:50.713624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.713416 5275 scope.go:122] "RemoveContainer" containerID="25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b" Nov 03 03:31:50.724744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.724719 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:31:50.730125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.730076 5275 scope.go:122] "RemoveContainer" containerID="25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b" Nov 03 03:31:50.730770 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:31:50.730744 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b\": container with ID starting with 25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b not found: ID does not exist" containerID="25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b" Nov 03 03:31:50.730903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.730781 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b"} err="failed to get container status \"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b\": rpc error: code = NotFound desc = could not find container \"25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b\": container with ID starting with 25ca3c99b07691dd76ade8596d68a605655d3ce455a246cca45086c275005e9b not found: ID does not exist" Nov 03 03:31:50.736695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.736674 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:31:50.749213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749170 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.749363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749262 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.749363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749298 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.749363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749340 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.749539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749443 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.749822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749776 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:50.749980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.749920 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750052 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750015 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.750123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750061 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750142 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750187 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750227 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.750345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750270 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750318 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.750508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750344 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.750508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750379 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.750508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750417 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.750883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750589 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750618 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750658 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\") pod \"80512c30-b976-4e03-93d3-f106775a5d85\" (UID: \"80512c30-b976-4e03-93d3-f106775a5d85\") " Nov 03 03:31:50.750883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.750791 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\") pod \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\" (UID: \"f9fd9bba-181d-4e48-bb2d-1585b2c39258\") " Nov 03 03:31:50.751572 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751546 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751578 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751594 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751608 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751623 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751639 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czv8l\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-kube-api-access-czv8l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751654 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80512c30-b976-4e03-93d3-f106775a5d85-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.751668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751667 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751681 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751694 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07bf80a-6769-4df6-9e65-bca09a30152a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751708 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g57st\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-kube-api-access-g57st\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751723 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751736 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751752 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.752090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751765 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.754967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.754944 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:50.755293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.755273 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.755698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.755453 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "kube-api-access-bkd4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.758394 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.758218 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:50.758664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.758516 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.758874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.758778 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.759224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.759080 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.759623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.759388 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.751779 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.760449 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.760468 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d850884-8836-4708-8b5a-b6fe51ee418d-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.760483 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d850884-8836-4708-8b5a-b6fe51ee418d-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.760510 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07bf80a-6769-4df6-9e65-bca09a30152a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.760677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.760524 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d850884-8836-4708-8b5a-b6fe51ee418d-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.763295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.763111 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.764223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.763495 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.764711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.764472 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:50.769258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.768128 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.770271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.770201 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:50.770271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.770233 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf" pod "f9fd9bba-181d-4e48-bb2d-1585b2c39258" (UID: "f9fd9bba-181d-4e48-bb2d-1585b2c39258"). InnerVolumeSpecName "kube-api-access-n7gjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.771166 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.771135 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token" pod "80512c30-b976-4e03-93d3-f106775a5d85" (UID: "80512c30-b976-4e03-93d3-f106775a5d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863095 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863135 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863152 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-kube-api-access-bkd4r\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863169 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80512c30-b976-4e03-93d3-f106775a5d85-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863183 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863197 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863212 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863226 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863240 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863253 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863267 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gjf\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-kube-api-access-n7gjf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863280 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9fd9bba-181d-4e48-bb2d-1585b2c39258-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863293 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863306 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80512c30-b976-4e03-93d3-f106775a5d85-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:50.863350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:50.863320 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80512c30-b976-4e03-93d3-f106775a5d85-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:51.068865 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.068655 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" podUID="07808c08-9603-428a-954b-3ab1b2d81801" containerName="payload-processing" containerID="cri-o://6cd9b9fe2bd19a9cafca739bca800f55961747c2739e6f6bd81cc18b6c1de7a1" gracePeriod=25 Nov 03 03:31:51.399409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.399338 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:31:51.422886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.420737 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-2z6xf"] Nov 03 03:31:51.601440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.601363 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:31:51.603034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.601970 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7vh27"] Nov 03 03:31:51.866036 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.865954 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" Nov 03 03:31:51.868070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.868041 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d" event={"ID":"f9fd9bba-181d-4e48-bb2d-1585b2c39258","Type":"ContainerDied","Data":"568c0b6da3b65eb426ba15f7f06c34439a84e2aa9a0ee615d378675556aec95b"} Nov 03 03:31:51.868493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.868474 5275 scope.go:122] "RemoveContainer" containerID="1c0d9984c8435d90364602fdb8b4642d30f2d05b07a404e454e6bc353525fb4e" Nov 03 03:31:51.874665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.874633 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" event={"ID":"80512c30-b976-4e03-93d3-f106775a5d85","Type":"ContainerDied","Data":"edc69b6c18bc38739a3cb774f45958803394a304093a362cc6e662236c6f21b3"} Nov 03 03:31:51.875192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.875173 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4" Nov 03 03:31:51.881210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.881185 5275 generic.go:356] "Generic (PLEG): container finished" podID="07808c08-9603-428a-954b-3ab1b2d81801" containerID="6cd9b9fe2bd19a9cafca739bca800f55961747c2739e6f6bd81cc18b6c1de7a1" exitCode=0 Nov 03 03:31:51.881469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.881382 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" event={"ID":"07808c08-9603-428a-954b-3ab1b2d81801","Type":"ContainerDied","Data":"6cd9b9fe2bd19a9cafca739bca800f55961747c2739e6f6bd81cc18b6c1de7a1"} Nov 03 03:31:51.891248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.891219 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:31:51.908691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:51.908667 5275 scope.go:122] "RemoveContainer" containerID="3991e9dc521e164e8929d45018879740ac98672d517335b9010e1b7e47b8e93c" Nov 03 03:31:52.019407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.019368 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\") pod \"07808c08-9603-428a-954b-3ab1b2d81801\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " Nov 03 03:31:52.019893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.019836 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume" pod "07808c08-9603-428a-954b-3ab1b2d81801" (UID: "07808c08-9603-428a-954b-3ab1b2d81801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:52.020119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.020096 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\" (UniqueName: \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\") pod \"07808c08-9603-428a-954b-3ab1b2d81801\" (UID: \"07808c08-9603-428a-954b-3ab1b2d81801\") " Nov 03 03:31:52.020834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.020790 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07808c08-9603-428a-954b-3ab1b2d81801-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:52.023146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.023115 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv" pod "07808c08-9603-428a-954b-3ab1b2d81801" (UID: "07808c08-9603-428a-954b-3ab1b2d81801"). InnerVolumeSpecName "kube-api-access-pg8lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:52.122715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.122614 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8lv\" (UniqueName: \"kubernetes.io/projected/07808c08-9603-428a-954b-3ab1b2d81801-kube-api-access-pg8lv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:52.170907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.170869 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:31:52.176929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.176892 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-z8d9d"] Nov 03 03:31:52.291304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.291239 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:31:52.301273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.301219 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5t2h4"] Nov 03 03:31:52.927175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.927138 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" event={"ID":"07808c08-9603-428a-954b-3ab1b2d81801","Type":"ContainerDied","Data":"3016c7b4402f770333ac4b292c7b52b56e9fefe74123f9919d5a1b856e09eea2"} Nov 03 03:31:52.927475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.927455 5275 scope.go:122] "RemoveContainer" containerID="6cd9b9fe2bd19a9cafca739bca800f55961747c2739e6f6bd81cc18b6c1de7a1" Nov 03 03:31:52.927709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:52.927693 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7" Nov 03 03:31:53.039560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.039520 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" path="/var/lib/kubelet/pods/7d850884-8836-4708-8b5a-b6fe51ee418d/volumes" Nov 03 03:31:53.040650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.040624 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80512c30-b976-4e03-93d3-f106775a5d85" path="/var/lib/kubelet/pods/80512c30-b976-4e03-93d3-f106775a5d85/volumes" Nov 03 03:31:53.041510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.041489 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" path="/var/lib/kubelet/pods/a07bf80a-6769-4df6-9e65-bca09a30152a/volumes" Nov 03 03:31:53.043519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.043497 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" path="/var/lib/kubelet/pods/f9fd9bba-181d-4e48-bb2d-1585b2c39258/volumes" Nov 03 03:31:53.149939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.149901 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:31:53.158872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.156414 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-398727-78567d589c-kmqj7"] Nov 03 03:31:53.532881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.529600 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-cjnn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:31:53.532881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:53.529672 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:31:54.356980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:54.356930 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:31:54.369973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:54.369932 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:31:54.985657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:54.985615 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07808c08-9603-428a-954b-3ab1b2d81801" path="/var/lib/kubelet/pods/07808c08-9603-428a-954b-3ab1b2d81801/volumes" Nov 03 03:31:58.052117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:58.052079 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw"] Nov 03 03:31:58.103049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:58.102984 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5085d947-xd7zw"] Nov 03 03:31:58.988122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:31:58.988091 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d34f5d-ad56-46a3-82cd-9201e72435e0" path="/var/lib/kubelet/pods/f7d34f5d-ad56-46a3-82cd-9201e72435e0/volumes" Nov 03 03:32:02.330840 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:32:02.330804 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:32:08.987286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987254 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:32:08.987827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987802 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987828 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987856 5275 cpu_manager.go:414] "containerMap: removing container" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987861 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987876 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987880 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" Nov 03 03:32:08.987895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987893 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987898 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987906 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987910 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987922 5275 cpu_manager.go:414] "containerMap: removing container" podUID="07808c08-9603-428a-954b-3ab1b2d81801" containerName="payload-processing" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987927 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07808c08-9603-428a-954b-3ab1b2d81801" containerName="payload-processing" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987941 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987946 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987953 5275 cpu_manager.go:414] "containerMap: removing container" podUID="df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" containerName="revoke-keys" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.987957 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" containerName="revoke-keys" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988049 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="80512c30-b976-4e03-93d3-f106775a5d85" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988060 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fd9bba-181d-4e48-bb2d-1585b2c39258" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988067 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988074 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7aa980-d3fc-4024-b511-642e6f7ff719" containerName="istio-proxy" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988080 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="07808c08-9603-428a-954b-3ab1b2d81801" containerName="payload-processing" Nov 03 03:32:08.988081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988086 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07bf80a-6769-4df6-9e65-bca09a30152a" containerName="istio-proxy" Nov 03 03:32:08.988532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988090 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" containerName="revoke-keys" Nov 03 03:32:08.988532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.988098 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d850884-8836-4708-8b5a-b6fe51ee418d" containerName="istio-proxy" Nov 03 03:32:08.991257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:08.991239 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.046777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.046747 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:32:09.092874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.092188 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4a5156\"" Nov 03 03:32:09.092874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.092312 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4a5156-dockercfg-28c6h\"" Nov 03 03:32:09.107911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.103537 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkt4m\" (UniqueName: \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.199220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.199168 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.221797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.221750 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:32:09.229135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.229107 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.230757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.230723 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:32:09.300938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.300900 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vl7\" (UniqueName: \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.301147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.301115 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.301283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.301264 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.301349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.301338 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkt4m\" (UniqueName: \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.301883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.301841 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.364663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.364633 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkt4m\" (UniqueName: \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.402503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.402471 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vl7\" (UniqueName: \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.402692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.402623 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.403357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.403336 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.463494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.463454 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vl7\" (UniqueName: \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\") pod \"payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.559550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.559512 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:09.605377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.605337 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:09.943661 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:09.942974 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:32:09.943661 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:32:09.943079 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1e8c71_5899_493d_b08f_db03309c5721.slice/crio-e9525297b1a29006b67a3ca3cfbb36aa3ee628d3bcccbec0d5b6ca86030ef78f WatchSource:0}: Error finding container e9525297b1a29006b67a3ca3cfbb36aa3ee628d3bcccbec0d5b6ca86030ef78f: Status 404 returned error can't find the container with id e9525297b1a29006b67a3ca3cfbb36aa3ee628d3bcccbec0d5b6ca86030ef78f Nov 03 03:32:10.043786 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:32:10.043740 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7086d91a_ce10_4ad0_93eb_35e87bb9e092.slice/crio-0700673090bcd13ba8833d038492e4813415cf48d57a1f14b95e82f41b912148 WatchSource:0}: Error finding container 0700673090bcd13ba8833d038492e4813415cf48d57a1f14b95e82f41b912148: Status 404 returned error can't find the container with id 0700673090bcd13ba8833d038492e4813415cf48d57a1f14b95e82f41b912148 Nov 03 03:32:10.044398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.044371 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:32:10.249989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.249878 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" event={"ID":"5f1e8c71-5899-493d-b08f-db03309c5721","Type":"ContainerStarted","Data":"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5"} Nov 03 03:32:10.249989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.249950 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" event={"ID":"5f1e8c71-5899-493d-b08f-db03309c5721","Type":"ContainerStarted","Data":"e9525297b1a29006b67a3ca3cfbb36aa3ee628d3bcccbec0d5b6ca86030ef78f"} Nov 03 03:32:10.249989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.249981 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:10.251718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.251686 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" event={"ID":"7086d91a-ce10-4ad0-93eb-35e87bb9e092","Type":"ContainerStarted","Data":"b8e417e216f5c3cee9b896702fa5eeda23870cec33c8bc99f01162906a528f30"} Nov 03 03:32:10.251870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.251745 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" event={"ID":"7086d91a-ce10-4ad0-93eb-35e87bb9e092","Type":"ContainerStarted","Data":"0700673090bcd13ba8833d038492e4813415cf48d57a1f14b95e82f41b912148"} Nov 03 03:32:10.251870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.251780 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:10.365969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:10.365909 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" podStartSLOduration=1.365892187 podStartE2EDuration="1.365892187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:10.365793781 +0000 UTC m=+1821.518482215" watchObservedRunningTime="2026-11-03 03:32:10.365892187 +0000 UTC m=+1821.518580609" Nov 03 03:32:21.263531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:21.263502 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:32:21.264020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:21.263587 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:32:21.384393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:21.384338 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" podStartSLOduration=13.384322687 podStartE2EDuration="13.384322687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:10.489381241 +0000 UTC m=+1821.642069664" watchObservedRunningTime="2026-11-03 03:32:21.384322687 +0000 UTC m=+1832.537011107" Nov 03 03:32:32.373736 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:32:32.373667 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:32:41.077981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:41.077943 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:32:41.078487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:41.078185 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" containerID="cri-o://245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277" gracePeriod=30 Nov 03 03:32:42.640600 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.640562 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:32:42.644887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.644867 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.712487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.700264 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-95ab58-openshift-default-dockercfg-f9dcb\"" Nov 03 03:32:42.758959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.758910 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:32:42.780658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.780619 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7c7\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.781029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.781000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.781189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.781174 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.781522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.781426 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.781724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.781705 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.781911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.781893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.782035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.782019 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.782162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.782148 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.782403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.782389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.884641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884599 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.884868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884699 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7c7\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.884868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884741 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.884868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884763 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.884868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884828 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884903 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884949 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.884971 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885009 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885261 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885427 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885622 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885678 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.885796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.885763 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.887582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.887554 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.888002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.887985 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.946832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.946759 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.947159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.947139 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7c7\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\") pod \"e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:42.967934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:42.967904 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:43.414943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.414911 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:32:43.415104 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:32:43.415079 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7069423_18e4_4e44_84de_76ae9bd70f38.slice/crio-8327f4bfd10d60a01bcd2217bc3b2b4f0e639eadbd339d1fcadb97aaebfd6e5b WatchSource:0}: Error finding container 8327f4bfd10d60a01bcd2217bc3b2b4f0e639eadbd339d1fcadb97aaebfd6e5b: Status 404 returned error can't find the container with id 8327f4bfd10d60a01bcd2217bc3b2b4f0e639eadbd339d1fcadb97aaebfd6e5b Nov 03 03:32:43.418525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.418486 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:32:43.418642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.418570 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:32:43.418642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.418614 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:32:43.550788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.550740 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerStarted","Data":"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db"} Nov 03 03:32:43.551013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.550809 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerStarted","Data":"8327f4bfd10d60a01bcd2217bc3b2b4f0e639eadbd339d1fcadb97aaebfd6e5b"} Nov 03 03:32:43.676384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.676257 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podStartSLOduration=1.676241308 podStartE2EDuration="1.676241308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:43.671766171 +0000 UTC m=+1854.824454666" watchObservedRunningTime="2026-11-03 03:32:43.676241308 +0000 UTC m=+1854.828929728" Nov 03 03:32:43.945557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.945474 5275 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:32:43.945557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.945529 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:32:43.968698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:43.968661 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:44.976789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:44.976755 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:45.566888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:45.566830 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:45.567988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:45.567966 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:32:46.372239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.372210 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:32:46.539532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.539499 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.539842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.539808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.540647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540624 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.540770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540663 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.540770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540699 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.541013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540796 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.541013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540939 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.541185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.541038 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.541185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.541123 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\") pod \"014fae9a-881d-4dc3-8209-45baf1b31029\" (UID: \"014fae9a-881d-4dc3-8209-45baf1b31029\") " Nov 03 03:32:46.542762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540242 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.542944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.540552 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:46.542944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.542772 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.543157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.543130 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.543574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.543548 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.546127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.546100 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:46.548841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.548807 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "kube-api-access-rs6tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:46.549454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.549433 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.555384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.555345 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo" pod "014fae9a-881d-4dc3-8209-45baf1b31029" (UID: "014fae9a-881d-4dc3-8209-45baf1b31029"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:32:46.580981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.580948 5275 generic.go:356] "Generic (PLEG): container finished" podID="014fae9a-881d-4dc3-8209-45baf1b31029" containerID="245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277" exitCode=0 Nov 03 03:32:46.581241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.581031 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" event={"ID":"014fae9a-881d-4dc3-8209-45baf1b31029","Type":"ContainerDied","Data":"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277"} Nov 03 03:32:46.581241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.581014 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" Nov 03 03:32:46.581241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.581094 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs" event={"ID":"014fae9a-881d-4dc3-8209-45baf1b31029","Type":"ContainerDied","Data":"6c1dad355a0c4100e106b737d5772165631efc1a91d41b001cc85bf27486f6ad"} Nov 03 03:32:46.581241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.581119 5275 scope.go:122] "RemoveContainer" containerID="245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277" Nov 03 03:32:46.595191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.595172 5275 scope.go:122] "RemoveContainer" containerID="245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277" Nov 03 03:32:46.595564 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:32:46.595539 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277\": container with ID starting with 245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277 not found: ID does not exist" containerID="245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277" Nov 03 03:32:46.595646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.595575 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277"} err="failed to get container status \"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277\": rpc error: code = NotFound desc = could not find container \"245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277\": container with ID starting with 245c197592b6dffce303206698e0b31dd4961407476af4b08d92505de2970277 not found: ID does not exist" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643470 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643506 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014fae9a-881d-4dc3-8209-45baf1b31029-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643519 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643529 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643538 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014fae9a-881d-4dc3-8209-45baf1b31029-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643546 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643556 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643565 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6tm\" (UniqueName: \"kubernetes.io/projected/014fae9a-881d-4dc3-8209-45baf1b31029-kube-api-access-rs6tm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.643975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.643574 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014fae9a-881d-4dc3-8209-45baf1b31029-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.753490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.753428 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:32:46.759543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.759510 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-xw9rs"] Nov 03 03:32:46.988127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:46.988046 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" path="/var/lib/kubelet/pods/014fae9a-881d-4dc3-8209-45baf1b31029/volumes" Nov 03 03:32:48.052436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:48.052398 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5"] Nov 03 03:32:48.100773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:48.100739 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b583a92-lgrd5"] Nov 03 03:32:48.988645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:48.988614 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd39aff-d32e-433b-8e2e-75040e135c1f" path="/var/lib/kubelet/pods/5cd39aff-d32e-433b-8e2e-75040e135c1f/volumes" Nov 03 03:32:50.175422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:50.175392 5275 scope.go:122] "RemoveContainer" containerID="4e2d577a068e1abdb8a29cafaf45b1e9429aab3abd2e4839183b696a08c563b6" Nov 03 03:32:50.188635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:50.188601 5275 scope.go:122] "RemoveContainer" containerID="5f0b9e1700f69447d6315ac629756470a2aa07467da2cac32b9585365353f3f8" Nov 03 03:32:50.198737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:32:50.198717 5275 scope.go:122] "RemoveContainer" containerID="f82bd16419b5254658e79a0761209f461451d68fd435770c3aafed14bb30a806" Nov 03 03:33:02.332504 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:33:02.332457 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:33:16.184575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:16.184542 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:33:16.185139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:16.184762 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" containerID="cri-o://28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd" gracePeriod=30 Nov 03 03:33:20.961151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:20.961112 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:33:20.961584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:20.961181 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:33:21.572159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.572119 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:33:21.756010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.755921 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756047 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756134 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756230 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756272 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756347 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756377 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756411 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.756524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.756459 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\") pod \"a071498d-79f4-40c0-b957-ed5a446f2858\" (UID: \"a071498d-79f4-40c0-b957-ed5a446f2858\") " Nov 03 03:33:21.757715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.757684 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:21.757975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.757932 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:21.758182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.758154 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:33:21.758260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.758225 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:21.758319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.758269 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:21.760015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.759958 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:21.769695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.769089 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:33:21.769695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.769572 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:21.769695 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.769665 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd" pod "a071498d-79f4-40c0-b957-ed5a446f2858" (UID: "a071498d-79f4-40c0-b957-ed5a446f2858"). InnerVolumeSpecName "kube-api-access-qpcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:21.859031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.858991 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpcgd\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-kube-api-access-qpcgd\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859027 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a071498d-79f4-40c0-b957-ed5a446f2858-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859042 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859056 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a071498d-79f4-40c0-b957-ed5a446f2858-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859071 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859086 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859100 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a071498d-79f4-40c0-b957-ed5a446f2858-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859112 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.859437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.859124 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a071498d-79f4-40c0-b957-ed5a446f2858-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:21.864283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.864253 5275 generic.go:356] "Generic (PLEG): container finished" podID="a071498d-79f4-40c0-b957-ed5a446f2858" containerID="28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd" exitCode=0 Nov 03 03:33:21.864430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.864328 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" Nov 03 03:33:21.864430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.864334 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" event={"ID":"a071498d-79f4-40c0-b957-ed5a446f2858","Type":"ContainerDied","Data":"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd"} Nov 03 03:33:21.864430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.864386 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr" event={"ID":"a071498d-79f4-40c0-b957-ed5a446f2858","Type":"ContainerDied","Data":"3e231c67a0da5ae4925d272ae452b372085e57e02af21086a08d1c9378c627e9"} Nov 03 03:33:21.864430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.864408 5275 scope.go:122] "RemoveContainer" containerID="28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd" Nov 03 03:33:21.878012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.877990 5275 scope.go:122] "RemoveContainer" containerID="28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd" Nov 03 03:33:21.878423 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:33:21.878397 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd\": container with ID starting with 28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd not found: ID does not exist" containerID="28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd" Nov 03 03:33:21.878651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:21.878623 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd"} err="failed to get container status \"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd\": rpc error: code = NotFound desc = could not find container \"28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd\": container with ID starting with 28cb5e0bd6578ecb708303ace4ed3a03ba63b38830eec384ae0bb501caaea1bd not found: ID does not exist" Nov 03 03:33:22.033499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:22.033463 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:33:22.039449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:22.039421 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-d6jwr"] Nov 03 03:33:22.986940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:22.986902 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" path="/var/lib/kubelet/pods/a071498d-79f4-40c0-b957-ed5a446f2858/volumes" Nov 03 03:33:32.331943 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:33:32.331901 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:33:35.261468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:35.261434 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:33:40.262354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.262288 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" podUID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" containerName="payload-pre-processing" containerID="cri-o://58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990" gracePeriod=25 Nov 03 03:33:40.619925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.619887 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:33:40.702061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.702023 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\" (UniqueName: \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\") pod \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " Nov 03 03:33:40.702256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.702106 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\") pod \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\" (UID: \"c8d55a1a-1c2a-48ab-b944-25e12c91a242\") " Nov 03 03:33:40.702547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.702517 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume" pod "c8d55a1a-1c2a-48ab-b944-25e12c91a242" (UID: "c8d55a1a-1c2a-48ab-b944-25e12c91a242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:33:40.702732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.702717 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d55a1a-1c2a-48ab-b944-25e12c91a242-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:40.704446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.704420 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4" pod "c8d55a1a-1c2a-48ab-b944-25e12c91a242" (UID: "c8d55a1a-1c2a-48ab-b944-25e12c91a242"). InnerVolumeSpecName "kube-api-access-tf9t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:40.803802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:40.803759 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9t4\" (UniqueName: \"kubernetes.io/projected/c8d55a1a-1c2a-48ab-b944-25e12c91a242-kube-api-access-tf9t4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:41.026923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.026878 5275 generic.go:356] "Generic (PLEG): container finished" podID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" containerID="58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990" exitCode=0 Nov 03 03:33:41.027104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.026939 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" Nov 03 03:33:41.027104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.026942 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" event={"ID":"c8d55a1a-1c2a-48ab-b944-25e12c91a242","Type":"ContainerDied","Data":"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990"} Nov 03 03:33:41.027104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.026999 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5" event={"ID":"c8d55a1a-1c2a-48ab-b944-25e12c91a242","Type":"ContainerDied","Data":"0597ae1a98935e4cb81ec9edf4a423330f4f7f9ddca7aabb5bdb9f2e1dca77de"} Nov 03 03:33:41.027104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.027029 5275 scope.go:122] "RemoveContainer" containerID="58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990" Nov 03 03:33:41.037409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.037388 5275 scope.go:122] "RemoveContainer" containerID="58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990" Nov 03 03:33:41.037744 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:33:41.037720 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990\": container with ID starting with 58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990 not found: ID does not exist" containerID="58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990" Nov 03 03:33:41.037796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.037769 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990"} err="failed to get container status \"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990\": rpc error: code = NotFound desc = could not find container \"58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990\": container with ID starting with 58ab4182fd9c4e7ba2aabdff30ec818daef9ba4d38685d58702360605bf11990 not found: ID does not exist" Nov 03 03:33:41.193915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.193879 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:33:41.197445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:41.197418 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-64c59d-598b54d6c5-l92g5"] Nov 03 03:33:42.986761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:42.986685 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" path="/var/lib/kubelet/pods/c8d55a1a-1c2a-48ab-b944-25e12c91a242/volumes" Nov 03 03:33:50.489793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:50.489754 5275 scope.go:122] "RemoveContainer" containerID="e764b42cf458c564f50850daa9d0b9da8cddfbff04ba957fe295b204062e5f9c" Nov 03 03:33:50.502433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:33:50.502409 5275 scope.go:122] "RemoveContainer" containerID="e2506994dff4a0020415cc4563cfd5094689d05479bd14a642518c4dc49e757d" Nov 03 03:34:02.337254 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:34:02.337216 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:34:07.073929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:07.073885 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq"] Nov 03 03:34:07.114541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:07.114509 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822a1b57-6zsnq"] Nov 03 03:34:08.336791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:08.336747 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:34:08.337322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:08.337008 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" containerID="cri-o://bd7312756456d39d8e6975792585c456bafcbcb39f14a19fbf65f45f188d73be" gracePeriod=30 Nov 03 03:34:08.935491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:08.935459 5275 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:34:08.935662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:08.935523 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:34:08.985903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:08.985831 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e094bab-ce26-45e5-abee-02f6efb4da51" path="/var/lib/kubelet/pods/0e094bab-ce26-45e5-abee-02f6efb4da51/volumes" Nov 03 03:34:13.584882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.581427 5275 generic.go:356] "Generic (PLEG): container finished" podID="77620089-d22c-4dbc-9916-d32bd6dde877" containerID="bd7312756456d39d8e6975792585c456bafcbcb39f14a19fbf65f45f188d73be" exitCode=0 Nov 03 03:34:13.584882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.581671 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" event={"ID":"77620089-d22c-4dbc-9916-d32bd6dde877","Type":"ContainerDied","Data":"bd7312756456d39d8e6975792585c456bafcbcb39f14a19fbf65f45f188d73be"} Nov 03 03:34:13.754004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.753969 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883302 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883406 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883450 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883482 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883590 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883626 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883665 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883734 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.885875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.883822 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\") pod \"77620089-d22c-4dbc-9916-d32bd6dde877\" (UID: \"77620089-d22c-4dbc-9916-d32bd6dde877\") " Nov 03 03:34:13.898274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.898193 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:13.916448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.916215 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:13.916880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.916682 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:13.917312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.917028 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:13.927432 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.927370 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:13.957172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.950767 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "kube-api-access-8q88f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:13.957172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.957036 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:13.957172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.957137 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:13.957617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.957574 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo" pod "77620089-d22c-4dbc-9916-d32bd6dde877" (UID: "77620089-d22c-4dbc-9916-d32bd6dde877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:34:13.994223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994188 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77620089-d22c-4dbc-9916-d32bd6dde877-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.994456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994436 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.994567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994554 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.994662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994648 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.994753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994741 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77620089-d22c-4dbc-9916-d32bd6dde877-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.994939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.994920 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q88f\" (UniqueName: \"kubernetes.io/projected/77620089-d22c-4dbc-9916-d32bd6dde877-kube-api-access-8q88f\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.995154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.995137 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.995266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.995253 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:13.995347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:13.995337 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77620089-d22c-4dbc-9916-d32bd6dde877-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:14.589829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:14.589793 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" Nov 03 03:34:14.590289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:14.589796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz" event={"ID":"77620089-d22c-4dbc-9916-d32bd6dde877","Type":"ContainerDied","Data":"6612a359488dee2efb8cde76d8b56f1778b5f047f8138faad178f71e662bf753"} Nov 03 03:34:14.590289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:14.589968 5275 scope.go:122] "RemoveContainer" containerID="bd7312756456d39d8e6975792585c456bafcbcb39f14a19fbf65f45f188d73be" Nov 03 03:34:14.828222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:14.828187 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:34:14.831994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:14.831969 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-4jctz"] Nov 03 03:34:15.136788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:15.136706 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" path="/var/lib/kubelet/pods/77620089-d22c-4dbc-9916-d32bd6dde877/volumes" Nov 03 03:34:17.417491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:17.417454 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:34:17.596953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:17.596914 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:34:22.417960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.417911 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" podUID="5f1e8c71-5899-493d-b08f-db03309c5721" containerName="payload-processing" containerID="cri-o://d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5" gracePeriod=25 Nov 03 03:34:22.598692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.598473 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" podUID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" containerName="payload-pre-processing" containerID="cri-o://b8e417e216f5c3cee9b896702fa5eeda23870cec33c8bc99f01162906a528f30" gracePeriod=25 Nov 03 03:34:22.725465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.725426 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:34:22.827452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.826665 5275 generic.go:356] "Generic (PLEG): container finished" podID="5f1e8c71-5899-493d-b08f-db03309c5721" containerID="d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5" exitCode=0 Nov 03 03:34:22.827452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.826776 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" event={"ID":"5f1e8c71-5899-493d-b08f-db03309c5721","Type":"ContainerDied","Data":"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5"} Nov 03 03:34:22.827452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.826872 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" event={"ID":"5f1e8c71-5899-493d-b08f-db03309c5721","Type":"ContainerDied","Data":"e9525297b1a29006b67a3ca3cfbb36aa3ee628d3bcccbec0d5b6ca86030ef78f"} Nov 03 03:34:22.827452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.826896 5275 scope.go:122] "RemoveContainer" containerID="d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5" Nov 03 03:34:22.827452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.826987 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n" Nov 03 03:34:22.833801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.833775 5275 generic.go:356] "Generic (PLEG): container finished" podID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" containerID="b8e417e216f5c3cee9b896702fa5eeda23870cec33c8bc99f01162906a528f30" exitCode=0 Nov 03 03:34:22.834226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.834074 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" event={"ID":"7086d91a-ce10-4ad0-93eb-35e87bb9e092","Type":"ContainerDied","Data":"b8e417e216f5c3cee9b896702fa5eeda23870cec33c8bc99f01162906a528f30"} Nov 03 03:34:22.842195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.842160 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\") pod \"5f1e8c71-5899-493d-b08f-db03309c5721\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " Nov 03 03:34:22.842661 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.842639 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\" (UniqueName: \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\") pod \"5f1e8c71-5899-493d-b08f-db03309c5721\" (UID: \"5f1e8c71-5899-493d-b08f-db03309c5721\") " Nov 03 03:34:22.844527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.844209 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume" pod "5f1e8c71-5899-493d-b08f-db03309c5721" (UID: "5f1e8c71-5899-493d-b08f-db03309c5721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:22.851558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.851529 5275 scope.go:122] "RemoveContainer" containerID="d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5" Nov 03 03:34:22.852077 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:34:22.852015 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5\": container with ID starting with d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5 not found: ID does not exist" containerID="d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5" Nov 03 03:34:22.852077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.852050 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5"} err="failed to get container status \"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5\": rpc error: code = NotFound desc = could not find container \"d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5\": container with ID starting with d094828f916db29bc954cb454b3cf775939c9b8323b74411d78fb538883b84c5 not found: ID does not exist" Nov 03 03:34:22.867446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.866024 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7" pod "5f1e8c71-5899-493d-b08f-db03309c5721" (UID: "5f1e8c71-5899-493d-b08f-db03309c5721"). InnerVolumeSpecName "kube-api-access-t4vl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:22.889473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.889436 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:34:22.944691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.944590 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\") pod \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " Nov 03 03:34:22.944932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.944796 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\" (UniqueName: \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\") pod \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\" (UID: \"7086d91a-ce10-4ad0-93eb-35e87bb9e092\") " Nov 03 03:34:22.945093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.945066 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume" pod "7086d91a-ce10-4ad0-93eb-35e87bb9e092" (UID: "7086d91a-ce10-4ad0-93eb-35e87bb9e092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:22.945378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.945360 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7086d91a-ce10-4ad0-93eb-35e87bb9e092-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:22.945464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.945379 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1e8c71-5899-493d-b08f-db03309c5721-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:22.945464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.945390 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vl7\" (UniqueName: \"kubernetes.io/projected/5f1e8c71-5899-493d-b08f-db03309c5721-kube-api-access-t4vl7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:22.947051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:22.947028 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m" pod "7086d91a-ce10-4ad0-93eb-35e87bb9e092" (UID: "7086d91a-ce10-4ad0-93eb-35e87bb9e092"). InnerVolumeSpecName "kube-api-access-kkt4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:23.046568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.046529 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkt4m\" (UniqueName: \"kubernetes.io/projected/7086d91a-ce10-4ad0-93eb-35e87bb9e092-kube-api-access-kkt4m\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:23.244235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.244135 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:34:23.247732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.247699 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-5557d9b86f-9x65n"] Nov 03 03:34:23.846093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.846056 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" Nov 03 03:34:23.846093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.846086 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf" event={"ID":"7086d91a-ce10-4ad0-93eb-35e87bb9e092","Type":"ContainerDied","Data":"0700673090bcd13ba8833d038492e4813415cf48d57a1f14b95e82f41b912148"} Nov 03 03:34:23.868424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:23.846152 5275 scope.go:122] "RemoveContainer" containerID="b8e417e216f5c3cee9b896702fa5eeda23870cec33c8bc99f01162906a528f30" Nov 03 03:34:24.012493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.012460 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:34:24.061334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.061296 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-6c95bfb4dc-pfccf"] Nov 03 03:34:24.105932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.105761 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:34:24.106112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.105985 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" containerID="cri-o://156b4eb2d5c22f2cab0ddb489722192d501a90ebfe6678f86dd609af4a7509c5" gracePeriod=30 Nov 03 03:34:24.855435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.855403 5275 generic.go:356] "Generic (PLEG): container finished" podID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerID="fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db" exitCode=0 Nov 03 03:34:24.855983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.855483 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerDied","Data":"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db"} Nov 03 03:34:24.856081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.856057 5275 scope.go:122] "RemoveContainer" containerID="fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db" Nov 03 03:34:24.857684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.857652 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:34:24.857788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.857722 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:34:24.857788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.857752 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:34:24.987080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.987046 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1e8c71-5899-493d-b08f-db03309c5721" path="/var/lib/kubelet/pods/5f1e8c71-5899-493d-b08f-db03309c5721/volumes" Nov 03 03:34:24.987616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:24.987594 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" path="/var/lib/kubelet/pods/7086d91a-ce10-4ad0-93eb-35e87bb9e092/volumes" Nov 03 03:34:25.881381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:25.881342 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerStarted","Data":"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1"} Nov 03 03:34:26.005979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:26.005944 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:34:26.009507 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:26.007395 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:26.009831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:26.009785 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:26.974273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:26.974226 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:26.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:26.974312 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:27.974341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:27.974299 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:27.974758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:27.974371 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:28.974076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:28.974011 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:28.974281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:28.974095 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:29.922679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.922327 5275 generic.go:356] "Generic (PLEG): container finished" podID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerID="156b4eb2d5c22f2cab0ddb489722192d501a90ebfe6678f86dd609af4a7509c5" exitCode=0 Nov 03 03:34:29.922679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.922442 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerDied","Data":"156b4eb2d5c22f2cab0ddb489722192d501a90ebfe6678f86dd609af4a7509c5"} Nov 03 03:34:29.922679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.922506 5275 scope.go:122] "RemoveContainer" containerID="b09dcbe2ca9e9d74e622fbe500939a748695dc8b59245b0c86361079cd23b261" Nov 03 03:34:29.971880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.970755 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:29.971880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.970812 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:29.995841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:29.995786 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:34:30.012545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012516 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012578 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012681 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012728 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012833 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012951 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.012995 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013085 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013120 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013173 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013298 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013321 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013377 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.013658 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\") pod \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\" (UID: \"8c76faf7-c4d1-4db5-9872-4ad2796ea655\") " Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.014519 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.014543 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.014558 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.014572 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.015462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.014586 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.016471 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.015804 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "kube-api-access-bnm9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:30.016471 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.016429 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:34:30.016934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.016910 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:30.017006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.016980 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo" pod "8c76faf7-c4d1-4db5-9872-4ad2796ea655" (UID: "8c76faf7-c4d1-4db5-9872-4ad2796ea655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:34:30.116160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.116122 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnm9v\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-kube-api-access-bnm9v\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.116160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.116156 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.116160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.116167 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.116405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.116176 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76faf7-c4d1-4db5-9872-4ad2796ea655-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:30.571007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.570963 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:34:30.931132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.931057 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" Nov 03 03:34:30.931528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.931053 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp" event={"ID":"8c76faf7-c4d1-4db5-9872-4ad2796ea655","Type":"ContainerDied","Data":"6e1de8ef37ea4bc71b9d578eb5c79a52aa0d8d62c28f059f04847ddc2b5d343f"} Nov 03 03:34:30.931528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.931203 5275 scope.go:122] "RemoveContainer" containerID="156b4eb2d5c22f2cab0ddb489722192d501a90ebfe6678f86dd609af4a7509c5" Nov 03 03:34:30.968610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.968571 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 03 03:34:30.968794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:30.968653 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 03 03:34:31.309931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:31.304790 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:34:31.384418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:31.373606 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-372186-openshift-default-75b9c87db7-4ctbp"] Nov 03 03:34:31.973562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:31.973523 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:34:31.974796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:31.974772 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:34:32.330729 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:34:32.330698 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:34:32.985104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:34:32.985074 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" path="/var/lib/kubelet/pods/8c76faf7-c4d1-4db5-9872-4ad2796ea655/volumes" Nov 03 03:35:02.331005 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:35:02.330967 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:35:08.934927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:08.934892 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:35:08.935474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:08.935128 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" containerID="cri-o://63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925" gracePeriod=30 Nov 03 03:35:12.049748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:12.049714 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n"] Nov 03 03:35:12.098977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:12.098939 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805d7ae6-l7c7n"] Nov 03 03:35:12.986525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:12.986485 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a914b79-2c67-48ca-8150-98d0030c326d" path="/var/lib/kubelet/pods/1a914b79-2c67-48ca-8150-98d0030c326d/volumes" Nov 03 03:35:14.220475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.220407 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:35:14.301031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.300992 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301042 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301093 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301109 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301258 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301313 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301341 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301407 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:14.301576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301416 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301479 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:14.301576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301531 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\") pod \"dea8dfcb-de12-4156-a668-d94a72673b92\" (UID: \"dea8dfcb-de12-4156-a668-d94a72673b92\") " Nov 03 03:35:14.301763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301590 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:14.301763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301599 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:14.301896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.301882 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:14.302187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.302157 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.302187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.302184 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea8dfcb-de12-4156-a668-d94a72673b92-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.302309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.302199 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.302309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.302211 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.302309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.302225 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.303527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.303503 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:14.303631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.303534 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:14.304277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.304255 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:35:14.304392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.304340 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s" pod "dea8dfcb-de12-4156-a668-d94a72673b92" (UID: "dea8dfcb-de12-4156-a668-d94a72673b92"). InnerVolumeSpecName "kube-api-access-qf42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:14.322840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.322809 5275 generic.go:356] "Generic (PLEG): container finished" podID="dea8dfcb-de12-4156-a668-d94a72673b92" containerID="63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925" exitCode=0 Nov 03 03:35:14.323025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.322892 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" Nov 03 03:35:14.323025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.322889 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" event={"ID":"dea8dfcb-de12-4156-a668-d94a72673b92","Type":"ContainerDied","Data":"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925"} Nov 03 03:35:14.323025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.323012 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp" event={"ID":"dea8dfcb-de12-4156-a668-d94a72673b92","Type":"ContainerDied","Data":"f351e3ff62b987aa2950d5a895c6485b47aa1df31cf86f8dd91ebff35ed2152f"} Nov 03 03:35:14.323198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.323040 5275 scope.go:122] "RemoveContainer" containerID="63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925" Nov 03 03:35:14.333472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.333454 5275 scope.go:122] "RemoveContainer" containerID="63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925" Nov 03 03:35:14.333786 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:35:14.333760 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925\": container with ID starting with 63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925 not found: ID does not exist" containerID="63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925" Nov 03 03:35:14.333835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.333796 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925"} err="failed to get container status \"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925\": rpc error: code = NotFound desc = could not find container \"63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925\": container with ID starting with 63ba0ee9f5708376e01453256c5e47812f746868a3306748e38328bdfbfa4925 not found: ID does not exist" Nov 03 03:35:14.404139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.404097 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf42s\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-kube-api-access-qf42s\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.404139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.404138 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea8dfcb-de12-4156-a668-d94a72673b92-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.404406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.404155 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea8dfcb-de12-4156-a668-d94a72673b92-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.404406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.404172 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea8dfcb-de12-4156-a668-d94a72673b92-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:14.506357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.506274 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:35:14.509134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.509113 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-hrvvp"] Nov 03 03:35:14.986758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:14.986723 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" path="/var/lib/kubelet/pods/dea8dfcb-de12-4156-a668-d94a72673b92/volumes" Nov 03 03:35:25.103861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.103800 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:35:25.104377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104361 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:25.104377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104377 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104415 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5f1e8c71-5899-493d-b08f-db03309c5721" containerName="payload-processing" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104421 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1e8c71-5899-493d-b08f-db03309c5721" containerName="payload-processing" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104434 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104439 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104447 5275 cpu_manager.go:414] "containerMap: removing container" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104451 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104458 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" containerName="payload-pre-processing" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104462 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" containerName="payload-pre-processing" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104469 5275 cpu_manager.go:414] "containerMap: removing container" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104473 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104489 5275 cpu_manager.go:414] "containerMap: removing container" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104493 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104510 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" containerName="payload-pre-processing" Nov 03 03:35:25.104523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104516 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" containerName="payload-pre-processing" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104594 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea8dfcb-de12-4156-a668-d94a72673b92" containerName="istio-proxy" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104605 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104612 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d55a1a-1c2a-48ab-b944-25e12c91a242" containerName="payload-pre-processing" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104619 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104628 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1e8c71-5899-493d-b08f-db03309c5721" containerName="payload-processing" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104633 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a071498d-79f4-40c0-b957-ed5a446f2858" containerName="istio-proxy" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104639 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="014fae9a-881d-4dc3-8209-45baf1b31029" containerName="istio-proxy" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104647 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7086d91a-ce10-4ad0-93eb-35e87bb9e092" containerName="payload-pre-processing" Nov 03 03:35:25.105516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.104656 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="77620089-d22c-4dbc-9916-d32bd6dde877" containerName="istio-proxy" Nov 03 03:35:25.109886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.109841 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173507 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173573 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173727 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173786 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173879 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.173959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.173942 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.174183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.174164 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.174266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.174249 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwd8\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.174349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.174327 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.193703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.193675 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a6b9ff-openshift-default-dockercfg-cx2hv\"" Nov 03 03:35:25.215611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.215572 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:35:25.291203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.291168 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.291946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.291919 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwd8\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.292520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.292498 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.297299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.297266 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.297840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.297810 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.298487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.298466 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.299032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.299013 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.299459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.299441 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.300061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.300041 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.300737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.299378 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.300894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.297725 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.300990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.297009 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.301096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.300412 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.301185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.291802 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.316238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.315251 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.322901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.320207 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.353264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.353227 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwd8\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.391975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.379380 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\") pod \"e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.423671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.423637 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:25.861390 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.861355 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:35:25.866884 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:35:25.866813 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45d272d_573e_47a1_a7a2_e596dfb41bef.slice/crio-9fbbee0bbed0e8fe69972dd3d8a5148acb9d930fc4f3904d43082009d50ec641 WatchSource:0}: Error finding container 9fbbee0bbed0e8fe69972dd3d8a5148acb9d930fc4f3904d43082009d50ec641: Status 404 returned error can't find the container with id 9fbbee0bbed0e8fe69972dd3d8a5148acb9d930fc4f3904d43082009d50ec641 Nov 03 03:35:25.869377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.869339 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:35:25.869464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.869413 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:35:25.869464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:25.869443 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:35:26.496572 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:26.496537 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerStarted","Data":"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a"} Nov 03 03:35:26.497026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:26.496591 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerStarted","Data":"9fbbee0bbed0e8fe69972dd3d8a5148acb9d930fc4f3904d43082009d50ec641"} Nov 03 03:35:26.619837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:26.619769 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podStartSLOduration=1.619753582 podStartE2EDuration="1.619753582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:26.616685687 +0000 UTC m=+2017.769374107" watchObservedRunningTime="2026-11-03 03:35:26.619753582 +0000 UTC m=+2017.772442002" Nov 03 03:35:27.424177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:27.424141 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:27.429993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:27.429963 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:27.520934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:27.519927 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:27.520934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:27.520915 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:35:32.331239 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:35:32.331204 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:35:34.079419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.079379 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:35:34.080998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.080972 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:34.081163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.081150 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c76faf7-c4d1-4db5-9872-4ad2796ea655" containerName="istio-proxy" Nov 03 03:35:34.117895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.117838 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.138931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.129784 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:35:34.175095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.175068 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a6b9ff\"" Nov 03 03:35:34.224158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.224118 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a6b9ff-dockercfg-8hc7k\"" Nov 03 03:35:34.302651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.302603 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.302901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.302665 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65h7k\" (UniqueName: \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.404204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.404103 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.404204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.404149 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65h7k\" (UniqueName: \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.404915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.404888 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.465081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.465046 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65h7k\" (UniqueName: \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\") pod \"payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.484480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.484441 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:34.862744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:34.862713 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:35:34.863690 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:35:34.863648 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48414c3_8f2b_4d5b_b1fc_f678f8468a7c.slice/crio-32cda516062029a7e018dbeffc7f6e27de832a930a5e0392b08613edbf5d9bc7 WatchSource:0}: Error finding container 32cda516062029a7e018dbeffc7f6e27de832a930a5e0392b08613edbf5d9bc7: Status 404 returned error can't find the container with id 32cda516062029a7e018dbeffc7f6e27de832a930a5e0392b08613edbf5d9bc7 Nov 03 03:35:35.585741 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:35.585705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" event={"ID":"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c","Type":"ContainerStarted","Data":"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194"} Nov 03 03:35:35.586293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:35.585763 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" event={"ID":"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c","Type":"ContainerStarted","Data":"32cda516062029a7e018dbeffc7f6e27de832a930a5e0392b08613edbf5d9bc7"} Nov 03 03:35:35.586293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:35.585792 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:35.743960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:35.743890 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" podStartSLOduration=1.743874763 podStartE2EDuration="1.743874763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:35.740710234 +0000 UTC m=+2026.893398656" watchObservedRunningTime="2026-11-03 03:35:35.743874763 +0000 UTC m=+2026.896563186" Nov 03 03:35:46.597468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:46.597436 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:35:50.585701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:50.585648 5275 scope.go:122] "RemoveContainer" containerID="c3e6a4197eac5b26ffda9066f1acfae89c74c53a84db9af3447b019d2589b1e9" Nov 03 03:35:50.604643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:35:50.604617 5275 scope.go:122] "RemoveContainer" containerID="a1a358a89bde5e31d5f28727e7071aff790c535208eb3055f8a2b916fd621310" Nov 03 03:36:02.331390 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:36:02.331349 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:36:13.323869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.323336 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:36:13.343815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.343782 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.405013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.404984 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d6e985-openshift-default-dockercfg-wn5gz\"" Nov 03 03:36:13.511594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.511538 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:36:13.528242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.528211 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.528583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.528515 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.528737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.528722 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.528981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.528964 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfq8d\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.529219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.529164 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.529580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.529558 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.529870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.529832 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.530086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.530031 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.530293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.530278 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.631632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.631551 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.631818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.631644 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.631818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.631754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.631818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.631784 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.631963 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632055 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfq8d\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632106 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632142 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632163 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632236 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632261 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632674 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.632778 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.632714 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.633084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.633030 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.634817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.634798 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.635079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.635062 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.693320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.693287 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.693495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.693470 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfq8d\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\") pod \"e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:13.981488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:13.981394 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:14.422510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.422486 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:36:14.424290 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:36:14.424266 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556ab9f8_6f92_4038_b0ef_15d4dc8228f3.slice/crio-09a32eb2256d1dacbaf554326666bdc10e1b8acbb81343a35fdc4a5ece7662bb WatchSource:0}: Error finding container 09a32eb2256d1dacbaf554326666bdc10e1b8acbb81343a35fdc4a5ece7662bb: Status 404 returned error can't find the container with id 09a32eb2256d1dacbaf554326666bdc10e1b8acbb81343a35fdc4a5ece7662bb Nov 03 03:36:14.426970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.426933 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:14.427056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.427002 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:14.427056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.427033 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:14.899128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.899083 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerStarted","Data":"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f"} Nov 03 03:36:14.899346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.899191 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerStarted","Data":"09a32eb2256d1dacbaf554326666bdc10e1b8acbb81343a35fdc4a5ece7662bb"} Nov 03 03:36:14.987002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:14.986970 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:15.103193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:15.103140 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podStartSLOduration=2.103125534 podStartE2EDuration="2.103125534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:15.069157019 +0000 UTC m=+2066.221845440" watchObservedRunningTime="2026-11-03 03:36:15.103125534 +0000 UTC m=+2066.255813954" Nov 03 03:36:15.989140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:15.989110 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:16.912590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:16.912558 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:16.913559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:16.913541 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:36:22.018542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.018501 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:36:22.024473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.024437 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.105873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.104984 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:36:22.141870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.141271 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6e985-dockercfg-xmssg\"" Nov 03 03:36:22.141870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.141404 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6e985\"" Nov 03 03:36:22.191763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.191706 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lq8l\" (UniqueName: \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.192000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.191925 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.293401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.293354 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lq8l\" (UniqueName: \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.293616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.293531 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.294241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.294221 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.354935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.354896 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lq8l\" (UniqueName: \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.364470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.364434 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:22.747706 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.747677 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:36:22.749604 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:36:22.749576 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0400fc12_9eb6_4cc6_be94_cb19130877b6.slice/crio-255a43eedf201be4c38f30bfb4d7de1cdbde3ba5066145523acb4725ba8e7be3 WatchSource:0}: Error finding container 255a43eedf201be4c38f30bfb4d7de1cdbde3ba5066145523acb4725ba8e7be3: Status 404 returned error can't find the container with id 255a43eedf201be4c38f30bfb4d7de1cdbde3ba5066145523acb4725ba8e7be3 Nov 03 03:36:22.966205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.966171 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" event={"ID":"0400fc12-9eb6-4cc6-be94-cb19130877b6","Type":"ContainerStarted","Data":"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903"} Nov 03 03:36:22.966404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.966222 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" event={"ID":"0400fc12-9eb6-4cc6-be94-cb19130877b6","Type":"ContainerStarted","Data":"255a43eedf201be4c38f30bfb4d7de1cdbde3ba5066145523acb4725ba8e7be3"} Nov 03 03:36:22.966404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:22.966311 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:23.082280 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:23.082195 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" podStartSLOduration=2.082176308 podStartE2EDuration="2.082176308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:23.081217775 +0000 UTC m=+2074.233906199" watchObservedRunningTime="2026-11-03 03:36:23.082176308 +0000 UTC m=+2074.234864729" Nov 03 03:36:32.333470 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:36:32.333410 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:36:33.992595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:33.992560 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:36:34.642532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.642496 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:36:34.646833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.646813 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.714966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.710484 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8200d4-openshift-default-dockercfg-h7gc8\"" Nov 03 03:36:34.741870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.735879 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805286 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805345 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805374 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805408 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805439 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805491 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlds\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805598 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805659 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.816870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.805754 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.909761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.909665 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.909761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.909742 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.909825 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.909959 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.909987 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.910012 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.910045 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.910079 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.910243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.910137 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlds\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.911099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.911072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.911809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.911780 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.912070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.912049 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.912310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.912269 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.912759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.912738 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.913929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.913909 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:34.915291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:34.915272 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:35.037912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.030725 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:35.045872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.039440 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlds\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\") pod \"e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:35.260887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.260766 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:35.771475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.771442 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:36:35.773673 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:36:35.773640 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661b866e_2e3b_4410_ac22_118ad3ea37f3.slice/crio-ef4dff109e272d7a22e163b0b5e568a1bac78a65514ec7451b8cdafc6496a296 WatchSource:0}: Error finding container ef4dff109e272d7a22e163b0b5e568a1bac78a65514ec7451b8cdafc6496a296: Status 404 returned error can't find the container with id ef4dff109e272d7a22e163b0b5e568a1bac78a65514ec7451b8cdafc6496a296 Nov 03 03:36:35.776574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.776534 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:35.776677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.776621 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:35.776677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:35.776668 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:36:36.099092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:36.099055 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerStarted","Data":"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70"} Nov 03 03:36:36.099606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:36.099120 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerStarted","Data":"ef4dff109e272d7a22e163b0b5e568a1bac78a65514ec7451b8cdafc6496a296"} Nov 03 03:36:36.221522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:36.221464 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podStartSLOduration=2.221445359 podStartE2EDuration="2.221445359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:36.21901391 +0000 UTC m=+2087.371702332" watchObservedRunningTime="2026-11-03 03:36:36.221445359 +0000 UTC m=+2087.374133781" Nov 03 03:36:36.261872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:36.261815 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:36.267690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:36.267661 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:37.107217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:37.107181 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:37.108407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:37.108383 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:36:39.051002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:39.050972 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8"] Nov 03 03:36:39.099807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:39.099778 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259d06d8-pw4g8"] Nov 03 03:36:39.545440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:39.545406 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:36:39.545647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:39.545622 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" containerID="cri-o://07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c" gracePeriod=30 Nov 03 03:36:40.986443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:40.986406 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df98ed0b-d5b4-413e-b0ea-c7d846ee14e7" path="/var/lib/kubelet/pods/df98ed0b-d5b4-413e-b0ea-c7d846ee14e7/volumes" Nov 03 03:36:42.854569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:42.854535 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:36:42.855100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:42.854613 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:36:44.848082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.848054 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:36:44.878823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.878775 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.878986 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879079 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879117 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879171 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879228 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879258 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.879478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.879454 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\") pod \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\" (UID: \"1d6ceaeb-f9a3-40f7-b907-c039719e5986\") " Nov 03 03:36:44.882230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.882182 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:44.882519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.882492 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:44.883315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.883270 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:44.883462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.883321 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:44.883462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.883375 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:44.883625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.883598 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:36:44.886218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.885313 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:44.892533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.891789 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:36:44.892533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.891931 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt" pod "1d6ceaeb-f9a3-40f7-b907-c039719e5986" (UID: "1d6ceaeb-f9a3-40f7-b907-c039719e5986"). InnerVolumeSpecName "kube-api-access-flnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981630 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981667 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981681 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981694 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981706 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnlt\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-kube-api-access-flnlt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981718 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981730 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981741 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ceaeb-f9a3-40f7-b907-c039719e5986-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:44.981779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:44.981754 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ceaeb-f9a3-40f7-b907-c039719e5986-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:45.172826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.172781 5275 generic.go:356] "Generic (PLEG): container finished" podID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerID="07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c" exitCode=0 Nov 03 03:36:45.173044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.172837 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" event={"ID":"1d6ceaeb-f9a3-40f7-b907-c039719e5986","Type":"ContainerDied","Data":"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c"} Nov 03 03:36:45.173044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.172914 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" Nov 03 03:36:45.173044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.172927 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44" event={"ID":"1d6ceaeb-f9a3-40f7-b907-c039719e5986","Type":"ContainerDied","Data":"6d4812bef66931a8968dce24137703a707e892e1a0370a936101883482b3085c"} Nov 03 03:36:45.173044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.172958 5275 scope.go:122] "RemoveContainer" containerID="07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c" Nov 03 03:36:45.184333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.184309 5275 scope.go:122] "RemoveContainer" containerID="07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c" Nov 03 03:36:45.184708 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:36:45.184678 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c\": container with ID starting with 07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c not found: ID does not exist" containerID="07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c" Nov 03 03:36:45.184800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.184721 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c"} err="failed to get container status \"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c\": rpc error: code = NotFound desc = could not find container \"07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c\": container with ID starting with 07eebcb177364ed63737cc3c6fb8e590bc1fb623035243d4850fa450bfd5a80c not found: ID does not exist" Nov 03 03:36:45.342292 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.342252 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:36:45.345961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:45.345932 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-wwc44"] Nov 03 03:36:46.988630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:46.988596 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" path="/var/lib/kubelet/pods/1d6ceaeb-f9a3-40f7-b907-c039719e5986/volumes" Nov 03 03:36:49.554984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:49.554954 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:36:49.560070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:49.560039 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:36:50.640157 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:36:50.640126 5275 scope.go:122] "RemoveContainer" containerID="d121ad33175c567ee19464df6979c8e64f361c5dc2872867f89de509d94681b1" Nov 03 03:37:02.330569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.330536 5275 generic.go:356] "Generic (PLEG): container finished" podID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerID="a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a" exitCode=0 Nov 03 03:37:02.331043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.330627 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerDied","Data":"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a"} Nov 03 03:37:02.331092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.331073 5275 scope.go:122] "RemoveContainer" containerID="a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a" Nov 03 03:37:02.333437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.333396 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:02.333545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.333498 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:02.333590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:02.333548 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:02.335002 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:02.334966 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:37:03.340267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:03.340229 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerStarted","Data":"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e"} Nov 03 03:37:03.424880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:03.424812 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:37:03.425071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:03.425019 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:03.425134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:03.425069 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:04.424785 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:04.424752 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:04.425213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:04.424809 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:05.424902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:05.424867 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:05.425397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:05.424924 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:06.424091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:06.424056 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:06.424283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:06.424113 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:07.424136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:07.424106 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:07.424522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:07.424163 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:08.424032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:08.423985 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 03:37:08.424307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:08.424285 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 03:37:09.429105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:09.429076 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:37:09.429603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:09.429582 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:37:09.430245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:09.430225 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:37:22.911034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.910997 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:37:22.911752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.911716 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" containerID="cri-o://4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b" gracePeriod=30 Nov 03 03:37:22.960619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.960567 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:37:22.960917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.960887 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" containerID="cri-o://a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2" gracePeriod=30 Nov 03 03:37:22.964836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.964799 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:37:22.965043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:22.965025 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" containerID="cri-o://540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952" gracePeriod=30 Nov 03 03:37:28.243426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.243390 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:37:28.302412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.302385 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:37:28.306241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.306217 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:37:28.366822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.366786 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.366830 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.366876 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.367005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.366897 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367194 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.367265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367228 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.367417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367388 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.367501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367433 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.367641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367516 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367548 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367584 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.367641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367643 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.367641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367682 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367731 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367788 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367823 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367898 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367927 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367956 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.367993 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368023 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.368037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368039 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368080 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368120 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368156 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368272 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\") pod \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\" (UID: \"8e2bfb15-325e-4909-b37b-cdc8338a1c09\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368310 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368360 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368419 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.368495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368455 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\") pod \"36ef7623-310e-4f3b-b24a-22d1410ba547\" (UID: \"36ef7623-310e-4f3b-b24a-22d1410ba547\") " Nov 03 03:37:28.368940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.368658 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\") pod \"7ce8feec-eabe-42cc-8638-07e5ab35da90\" (UID: \"7ce8feec-eabe-42cc-8638-07e5ab35da90\") " Nov 03 03:37:28.369414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369015 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.369414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369131 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.369414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369166 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:28.369414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369405 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369566 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369584 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369590 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369599 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369635 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.369672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369672 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef7623-310e-4f3b-b24a-22d1410ba547-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369686 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369700 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369715 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369868 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369892 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "kube-api-access-fwnpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.370012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.369997 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.370325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.370108 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.370325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.370135 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.370703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.370594 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:28.370837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.370698 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.371258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.371226 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.371258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.371245 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "kube-api-access-cpk24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.372181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.372154 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:37:28.373024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.372992 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.373152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.373128 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.373346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.373326 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:37:28.373438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.373421 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.373680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.373650 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy" pod "36ef7623-310e-4f3b-b24a-22d1410ba547" (UID: "36ef7623-310e-4f3b-b24a-22d1410ba547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.374446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.374410 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "kube-api-access-g8w64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:28.374446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.374435 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy" pod "7ce8feec-eabe-42cc-8638-07e5ab35da90" (UID: "7ce8feec-eabe-42cc-8638-07e5ab35da90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:28.375115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.375093 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo" pod "8e2bfb15-325e-4909-b37b-cdc8338a1c09" (UID: "8e2bfb15-325e-4909-b37b-cdc8338a1c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471524 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471555 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471564 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471572 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471581 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471589 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8w64\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-kube-api-access-g8w64\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471598 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce8feec-eabe-42cc-8638-07e5ab35da90-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.471606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471606 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471616 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471625 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471633 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471645 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef7623-310e-4f3b-b24a-22d1410ba547-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471654 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce8feec-eabe-42cc-8638-07e5ab35da90-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471662 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnpx\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-kube-api-access-fwnpx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471673 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpk24\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-kube-api-access-cpk24\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471681 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef7623-310e-4f3b-b24a-22d1410ba547-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471690 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471698 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce8feec-eabe-42cc-8638-07e5ab35da90-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.472000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.471706 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2bfb15-325e-4909-b37b-cdc8338a1c09-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:28.622501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.622467 5275 generic.go:356] "Generic (PLEG): container finished" podID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerID="a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2" exitCode=0 Nov 03 03:37:28.622704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.622521 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" Nov 03 03:37:28.622704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.622552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" event={"ID":"7ce8feec-eabe-42cc-8638-07e5ab35da90","Type":"ContainerDied","Data":"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2"} Nov 03 03:37:28.622704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.622604 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn" event={"ID":"7ce8feec-eabe-42cc-8638-07e5ab35da90","Type":"ContainerDied","Data":"aeb7617b61eb9a7b611b95df5b111a42206245c33de0f750cf6426ea39ee450e"} Nov 03 03:37:28.622704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.622631 5275 scope.go:122] "RemoveContainer" containerID="a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2" Nov 03 03:37:28.624414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.624390 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerID="540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952" exitCode=0 Nov 03 03:37:28.624517 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.624456 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" Nov 03 03:37:28.624592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.624511 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerDied","Data":"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952"} Nov 03 03:37:28.624592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.624563 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9" event={"ID":"8e2bfb15-325e-4909-b37b-cdc8338a1c09","Type":"ContainerDied","Data":"4aa2572f079bfd62a177f042bc189109c47427918ec78f1ae4d19858881022c3"} Nov 03 03:37:28.626187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.626158 5275 generic.go:356] "Generic (PLEG): container finished" podID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerID="4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b" exitCode=0 Nov 03 03:37:28.626295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.626184 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" event={"ID":"36ef7623-310e-4f3b-b24a-22d1410ba547","Type":"ContainerDied","Data":"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b"} Nov 03 03:37:28.626295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.626204 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" Nov 03 03:37:28.626295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.626233 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b" event={"ID":"36ef7623-310e-4f3b-b24a-22d1410ba547","Type":"ContainerDied","Data":"d4d715d5fe23725e7c974234abfa8f32082fa6791ce2f80cffe7e558b1273bb3"} Nov 03 03:37:28.636001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.635986 5275 scope.go:122] "RemoveContainer" containerID="a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2" Nov 03 03:37:28.656899 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:28.643057 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2\": container with ID starting with a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2 not found: ID does not exist" containerID="a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2" Nov 03 03:37:28.656899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.643092 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2"} err="failed to get container status \"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2\": rpc error: code = NotFound desc = could not find container \"a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2\": container with ID starting with a3b6d72ad3977eaed4b739e44e3723e4edbcd516db2179eeecb32cbaaa731bb2 not found: ID does not exist" Nov 03 03:37:28.656899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.643119 5275 scope.go:122] "RemoveContainer" containerID="540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952" Nov 03 03:37:28.673317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.672916 5275 scope.go:122] "RemoveContainer" containerID="85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb" Nov 03 03:37:28.686065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.686040 5275 scope.go:122] "RemoveContainer" containerID="540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952" Nov 03 03:37:28.686440 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:28.686405 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952\": container with ID starting with 540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952 not found: ID does not exist" containerID="540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952" Nov 03 03:37:28.686538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.686457 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952"} err="failed to get container status \"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952\": rpc error: code = NotFound desc = could not find container \"540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952\": container with ID starting with 540b55945fde080d0f86c13516237fd858e518f2044368c5f7d050c45c977952 not found: ID does not exist" Nov 03 03:37:28.686538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.686482 5275 scope.go:122] "RemoveContainer" containerID="85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb" Nov 03 03:37:28.686917 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:28.686880 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb\": container with ID starting with 85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb not found: ID does not exist" containerID="85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb" Nov 03 03:37:28.687039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.686918 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb"} err="failed to get container status \"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb\": rpc error: code = NotFound desc = could not find container \"85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb\": container with ID starting with 85d2682929428d488c2b3504c35dba9618e9c3d939b316fa5ec3f95bbb39d5bb not found: ID does not exist" Nov 03 03:37:28.687039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.686942 5275 scope.go:122] "RemoveContainer" containerID="4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b" Nov 03 03:37:28.697676 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.697655 5275 scope.go:122] "RemoveContainer" containerID="4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b" Nov 03 03:37:28.698030 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:28.698006 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b\": container with ID starting with 4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b not found: ID does not exist" containerID="4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b" Nov 03 03:37:28.698122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.698039 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b"} err="failed to get container status \"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b\": rpc error: code = NotFound desc = could not find container \"4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b\": container with ID starting with 4461e3907b6e1b1ee75c33f3cf2ddbf9bd027fffdec659ed5f4ed3651e93fa1b not found: ID does not exist" Nov 03 03:37:28.774788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.774716 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:37:28.778054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.778034 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6948b"] Nov 03 03:37:28.883287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.883247 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:37:28.889771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.889742 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-cjnn9"] Nov 03 03:37:28.985051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.985015 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" path="/var/lib/kubelet/pods/36ef7623-310e-4f3b-b24a-22d1410ba547/volumes" Nov 03 03:37:28.985472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.985458 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" path="/var/lib/kubelet/pods/8e2bfb15-325e-4909-b37b-cdc8338a1c09/volumes" Nov 03 03:37:28.997646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:28.997616 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:37:29.003980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:29.003954 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-skdrn"] Nov 03 03:37:30.988025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:30.987990 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" path="/var/lib/kubelet/pods/7ce8feec-eabe-42cc-8638-07e5ab35da90/volumes" Nov 03 03:37:32.331228 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:32.331194 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:37:43.001897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.001861 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:37:43.002655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002631 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002657 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002687 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002694 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002713 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002720 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002739 5275 cpu_manager.go:414] "containerMap: removing container" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" Nov 03 03:37:43.002754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002746 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002772 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002779 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002954 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6ceaeb-f9a3-40f7-b907-c039719e5986" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002974 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002985 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2bfb15-325e-4909-b37b-cdc8338a1c09" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.002996 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ef7623-310e-4f3b-b24a-22d1410ba547" containerName="istio-proxy" Nov 03 03:37:43.003129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.003007 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce8feec-eabe-42cc-8638-07e5ab35da90" containerName="istio-proxy" Nov 03 03:37:43.006481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.006458 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.063607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.063570 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:37:43.102840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.102793 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.103091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.103069 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vp5h\" (UniqueName: \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.108494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.108459 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-160d39\"" Nov 03 03:37:43.108681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.108664 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-160d39-dockercfg-jwjdx\"" Nov 03 03:37:43.204170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.204131 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.204373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.204281 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vp5h\" (UniqueName: \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.204881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.204826 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.265360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.265284 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vp5h\" (UniqueName: \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\") pod \"payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.320608 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.320563 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:43.706748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.706717 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:37:43.707631 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:37:43.707598 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b57a00_14c8_4fd1_bf47_cadbabe91a26.slice/crio-2ee525717c860af656c346b88d7db874e8aaf35fc79e56026a987834eefa0006 WatchSource:0}: Error finding container 2ee525717c860af656c346b88d7db874e8aaf35fc79e56026a987834eefa0006: Status 404 returned error can't find the container with id 2ee525717c860af656c346b88d7db874e8aaf35fc79e56026a987834eefa0006 Nov 03 03:37:43.709544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.709527 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:37:43.735803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.735772 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" event={"ID":"25b57a00-14c8-4fd1-bf47-cadbabe91a26","Type":"ContainerStarted","Data":"2ee525717c860af656c346b88d7db874e8aaf35fc79e56026a987834eefa0006"} Nov 03 03:37:43.798724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.798692 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:37:43.804225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.804186 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:43.863544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.863503 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:37:43.913833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.913796 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:43.914029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:43.913876 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfk6\" (UniqueName: \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.015772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.015692 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.015772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.015739 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfk6\" (UniqueName: \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.016433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.016411 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.079760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.079725 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfk6\" (UniqueName: \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\") pod \"payload-processing-e2e-ipp-a-160d39-656b74948-742fx\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.118675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.118642 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:44.499334 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:37:44.499302 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f0341e_c8f4_4941_bf8c_8721bcd1277b.slice/crio-cdff35d215f09dadcd5bbcda1f36dcdfdffb93b52e4b103cccf2b4d3fe03ee1f WatchSource:0}: Error finding container cdff35d215f09dadcd5bbcda1f36dcdfdffb93b52e4b103cccf2b4d3fe03ee1f: Status 404 returned error can't find the container with id cdff35d215f09dadcd5bbcda1f36dcdfdffb93b52e4b103cccf2b4d3fe03ee1f Nov 03 03:37:44.500284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.500163 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:37:44.746182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:44.746084 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" event={"ID":"d9f0341e-c8f4-4941-bf8c-8721bcd1277b","Type":"ContainerStarted","Data":"cdff35d215f09dadcd5bbcda1f36dcdfdffb93b52e4b103cccf2b4d3fe03ee1f"} Nov 03 03:37:46.844873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:46.844401 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" event={"ID":"d9f0341e-c8f4-4941-bf8c-8721bcd1277b","Type":"ContainerStarted","Data":"71f4c57ac2a4bcdd415cceece57fb8a3498a1f169993ebc08f8766e866810e5e"} Nov 03 03:37:46.848870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:46.845568 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:46.858883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:46.857315 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" event={"ID":"25b57a00-14c8-4fd1-bf47-cadbabe91a26","Type":"ContainerStarted","Data":"28c08fe7ed15a1f9bd353ed730e511cb8991dfae9661e76fc61bfa7e5e058536"} Nov 03 03:37:46.858883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:46.858048 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:47.139030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:47.138905 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" podStartSLOduration=2.7124264829999998 podStartE2EDuration="4.138885347s" totalImagesPullingTime="1.426458864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:47.018264714 +0000 UTC m=+2158.170953136" watchObservedRunningTime="2026-11-03 03:37:47.138885347 +0000 UTC m=+2158.291573769" Nov 03 03:37:47.140685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:47.140634 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" podStartSLOduration=3.099077751 podStartE2EDuration="5.140621643s" totalImagesPullingTime="2.041543892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:47.135555377 +0000 UTC m=+2158.288243802" watchObservedRunningTime="2026-11-03 03:37:47.140621643 +0000 UTC m=+2158.293310066" Nov 03 03:37:50.678018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:50.677978 5275 scope.go:122] "RemoveContainer" containerID="4131397ef451deb3e0efcaf858d3435f150058e607fd97b824f7d2dbdecf0b68" Nov 03 03:37:54.123278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.123192 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:37:54.566561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.566515 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:37:54.573770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.573742 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.632789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.631085 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b9c10a-openshift-default-dockercfg-gtf4m\"" Nov 03 03:37:54.634109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.633903 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.728744 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.728825 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.728909 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv99r\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.728954 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.728999 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.729074 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.729111 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.729195 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.742870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.729256 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.831525 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.831610 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.831670 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv99r\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.831727 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.831808 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832022 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832096 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832202 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832253 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832675 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832884 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.832977 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.833224 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.833470 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.835905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.835706 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.841410 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.841382 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.894193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.894155 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv99r\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:54.895323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:54.895296 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\") pod \"e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:55.192756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.192663 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:55.653874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.653181 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:55.653874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.653270 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:55.653874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.653316 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:55.656423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.656394 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:37:55.949928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.949318 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerStarted","Data":"471e7492ac7a70b8daf3b6abf727aaa26a771be9898c144cd41e283ca0dbeed3"} Nov 03 03:37:55.949928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:55.949419 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerStarted","Data":"5df1a33b64321bddc55ea78eab64cc2e46ce593fec1f8715082e57fe9f695f27"} Nov 03 03:37:56.080428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:56.080365 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podStartSLOduration=2.080343965 podStartE2EDuration="2.080343965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:56.071233984 +0000 UTC m=+2167.223922406" watchObservedRunningTime="2026-11-03 03:37:56.080343965 +0000 UTC m=+2167.233032387" Nov 03 03:37:56.197871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:56.196651 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:56.202973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:56.198709 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:37:56.202973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:56.198781 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:37:57.198414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:57.198382 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:57.967291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:57.967256 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:57.968550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:57.968524 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:37:58.874082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.873992 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:37:58.874579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.874463 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:37:58.977897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.977772 5275 generic.go:356] "Generic (PLEG): container finished" podID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerID="bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f" exitCode=0 Nov 03 03:37:58.986347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.986312 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerDied","Data":"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f"} Nov 03 03:37:58.986789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.986768 5275 scope.go:122] "RemoveContainer" containerID="bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f" Nov 03 03:37:58.988562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.988530 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:58.988688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.988620 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:58.988688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:58.988672 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:37:59.123647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.123608 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" podUID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" containerName="payload-processing" containerID="cri-o://078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194" gracePeriod=25 Nov 03 03:37:59.479072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.479045 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:37:59.523895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.523834 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\" (UniqueName: \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\") pod \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " Nov 03 03:37:59.524105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.523978 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\") pod \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\" (UID: \"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c\") " Nov 03 03:37:59.524420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.524388 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume" pod "e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" (UID: "e48414c3-8f2b-4d5b-b1fc-f678f8468a7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:59.524637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.524618 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:59.526668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.526643 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k" pod "e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" (UID: "e48414c3-8f2b-4d5b-b1fc-f678f8468a7c"). InnerVolumeSpecName "kube-api-access-65h7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:59.629095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.627973 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65h7k\" (UniqueName: \"kubernetes.io/projected/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c-kube-api-access-65h7k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:59.986692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.986606 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerStarted","Data":"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b"} Nov 03 03:37:59.988130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.988100 5275 generic.go:356] "Generic (PLEG): container finished" podID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" containerID="078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194" exitCode=0 Nov 03 03:37:59.988258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.988138 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" Nov 03 03:37:59.988258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.988171 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" event={"ID":"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c","Type":"ContainerDied","Data":"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194"} Nov 03 03:37:59.988258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.988223 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv" event={"ID":"e48414c3-8f2b-4d5b-b1fc-f678f8468a7c","Type":"ContainerDied","Data":"32cda516062029a7e018dbeffc7f6e27de832a930a5e0392b08613edbf5d9bc7"} Nov 03 03:37:59.988258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.988247 5275 scope.go:122] "RemoveContainer" containerID="078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194" Nov 03 03:37:59.999081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.999060 5275 scope.go:122] "RemoveContainer" containerID="078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194" Nov 03 03:37:59.999378 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:37:59.999356 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194\": container with ID starting with 078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194 not found: ID does not exist" containerID="078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194" Nov 03 03:37:59.999431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:37:59.999388 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194"} err="failed to get container status \"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194\": rpc error: code = NotFound desc = could not find container \"078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194\": container with ID starting with 078e4328bd7ec73908e5e9b00486fe96bcb006eab62a287d3938df4a53c0a194 not found: ID does not exist" Nov 03 03:38:00.236270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.236226 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:38:00.243461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.243377 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-58b6b86669-rsrdv"] Nov 03 03:38:00.983291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.983254 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 03:38:00.983487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.983330 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 03:38:00.986491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.986461 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" path="/var/lib/kubelet/pods/e48414c3-8f2b-4d5b-b1fc-f678f8468a7c/volumes" Nov 03 03:38:00.986791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:00.986777 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:38:01.912727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:01.912689 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:38:01.981711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:01.981675 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 03:38:01.981926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:01.981739 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 03:38:02.330623 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:38:02.330587 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:38:02.982877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:02.982812 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 03:38:02.983067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:02.982898 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 03:38:03.982868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:03.982317 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 03:38:03.982868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:03.982379 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 03:38:04.109875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.102977 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:38:04.109875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.103925 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" containerName="payload-processing" Nov 03 03:38:04.109875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.103946 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" containerName="payload-processing" Nov 03 03:38:04.109875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.104116 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48414c3-8f2b-4d5b-b1fc-f678f8468a7c" containerName="payload-processing" Nov 03 03:38:04.125907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.124757 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:38:04.125907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.124972 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.147552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.147507 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:38:04.194944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.194912 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b9c10a\"" Nov 03 03:38:04.204445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.204420 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.213167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.213078 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:38:04.281792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.281615 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b9c10a-dockercfg-66s7h\"" Nov 03 03:38:04.285994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.284278 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.286173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.286073 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d975\" (UniqueName: \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.286246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.286197 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.286391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.286370 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6x2\" (UniqueName: \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.387391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.387354 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.387591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.387444 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d975\" (UniqueName: \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.387591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.387556 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.387714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.387634 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6x2\" (UniqueName: \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.388409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.388383 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.388771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.388753 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.467872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.467551 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d975\" (UniqueName: \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\") pod \"payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.485875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.484775 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6x2\" (UniqueName: \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.503663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.503621 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:04.522686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.522653 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:04.924974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.924948 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:38:04.926736 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:38:04.926712 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ece53d_af9a_4022_acd6_10084d8aed6b.slice/crio-adbebbedb2df2e1570cb0f6d2a5a967a8eb49dea4083c928e588214ae51c0eda WatchSource:0}: Error finding container adbebbedb2df2e1570cb0f6d2a5a967a8eb49dea4083c928e588214ae51c0eda: Status 404 returned error can't find the container with id adbebbedb2df2e1570cb0f6d2a5a967a8eb49dea4083c928e588214ae51c0eda Nov 03 03:38:04.982379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.982326 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 03:38:04.982557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.982406 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 03:38:04.990823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:04.990799 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:38:05.109730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.109693 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" event={"ID":"945ce218-284d-4d5d-94df-eb45e0ccb44b","Type":"ContainerStarted","Data":"f1b467a0b964f5f5a46edefecf423cbad62ecad48c2a6e53a487ebb229b47f10"} Nov 03 03:38:05.109932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.109759 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" event={"ID":"945ce218-284d-4d5d-94df-eb45e0ccb44b","Type":"ContainerStarted","Data":"3838f9398ad839273018692c08780f53bc1a2fd8bc30b72beaace7834e44e84d"} Nov 03 03:38:05.109932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.109788 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:05.111567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.111541 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" event={"ID":"f9ece53d-af9a-4022-acd6-10084d8aed6b","Type":"ContainerStarted","Data":"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900"} Nov 03 03:38:05.111711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.111588 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:05.111711 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.111605 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" event={"ID":"f9ece53d-af9a-4022-acd6-10084d8aed6b","Type":"ContainerStarted","Data":"adbebbedb2df2e1570cb0f6d2a5a967a8eb49dea4083c928e588214ae51c0eda"} Nov 03 03:38:05.228936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.228786 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" podStartSLOduration=1.228771706 podStartE2EDuration="1.228771706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:05.225376896 +0000 UTC m=+2176.378065322" watchObservedRunningTime="2026-11-03 03:38:05.228771706 +0000 UTC m=+2176.381460127" Nov 03 03:38:05.352118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.348792 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" podStartSLOduration=2.3487663100000002 podStartE2EDuration="2.34876631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:05.342730867 +0000 UTC m=+2176.495419290" watchObservedRunningTime="2026-11-03 03:38:05.34876631 +0000 UTC m=+2176.501454808" Nov 03 03:38:05.988449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.988402 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:38:05.989662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:05.989635 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:38:16.120736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:16.120702 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:38:16.121313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:16.120780 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:38:23.159873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.157487 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:38:23.174876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.171149 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.213809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.213779 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:38:23.272260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.272220 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6ef2c\"" Nov 03 03:38:23.275537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.272713 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6ef2c-dockercfg-zstps\"" Nov 03 03:38:23.285131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.285100 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.285598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.285576 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58wc\" (UniqueName: \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.386529 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.386490 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.386764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.386732 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58wc\" (UniqueName: \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.387598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.387572 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.481674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.481574 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58wc\" (UniqueName: \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.491206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.491179 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:23.888465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.888439 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:38:23.889949 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:38:23.889926 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64fd76f_5f62_4994_87cd_3389d6b3319f.slice/crio-27640054cec1526ece01643b702d90b6ce9c1ff123b551dd09c754a737f73d33 WatchSource:0}: Error finding container 27640054cec1526ece01643b702d90b6ce9c1ff123b551dd09c754a737f73d33: Status 404 returned error can't find the container with id 27640054cec1526ece01643b702d90b6ce9c1ff123b551dd09c754a737f73d33 Nov 03 03:38:23.938407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.938371 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:38:23.966887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:23.948322 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.006784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.006535 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:38:24.096133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.096103 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9d8j\" (UniqueName: \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.096332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.096194 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.198401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.198312 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.198966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.198603 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9d8j\" (UniqueName: \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.199308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.199278 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.277685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.277585 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9d8j\" (UniqueName: \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\") pod \"payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.284900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.284861 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" event={"ID":"b64fd76f-5f62-4994-87cd-3389d6b3319f","Type":"ContainerStarted","Data":"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476"} Nov 03 03:38:24.285067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.284922 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" event={"ID":"b64fd76f-5f62-4994-87cd-3389d6b3319f","Type":"ContainerStarted","Data":"27640054cec1526ece01643b702d90b6ce9c1ff123b551dd09c754a737f73d33"} Nov 03 03:38:24.285123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.285089 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:24.301579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.301546 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:24.423881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.423199 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" podStartSLOduration=1.423176719 podStartE2EDuration="1.423176719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:24.415645507 +0000 UTC m=+2195.568333929" watchObservedRunningTime="2026-11-03 03:38:24.423176719 +0000 UTC m=+2195.575865143" Nov 03 03:38:24.722522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:24.722490 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:38:24.723354 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:38:24.723321 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32d82b3_c0e1_4298_bdb1_e58b37af8639.slice/crio-dc3a11c33ff25bb4a6be07cacda509c361109f5e7dd404cdc84985de6500447a WatchSource:0}: Error finding container dc3a11c33ff25bb4a6be07cacda509c361109f5e7dd404cdc84985de6500447a: Status 404 returned error can't find the container with id dc3a11c33ff25bb4a6be07cacda509c361109f5e7dd404cdc84985de6500447a Nov 03 03:38:25.297500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:25.297462 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" event={"ID":"e32d82b3-c0e1-4298-bdb1-e58b37af8639","Type":"ContainerStarted","Data":"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d"} Nov 03 03:38:25.298097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:25.297520 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" event={"ID":"e32d82b3-c0e1-4298-bdb1-e58b37af8639","Type":"ContainerStarted","Data":"dc3a11c33ff25bb4a6be07cacda509c361109f5e7dd404cdc84985de6500447a"} Nov 03 03:38:25.298097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:25.297559 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:25.420340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:25.420262 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" podStartSLOduration=2.420243171 podStartE2EDuration="2.420243171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:38:25.413976865 +0000 UTC m=+2196.566665294" watchObservedRunningTime="2026-11-03 03:38:25.420243171 +0000 UTC m=+2196.572931592" Nov 03 03:38:29.333183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.333151 5275 generic.go:356] "Generic (PLEG): container finished" podID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerID="32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70" exitCode=0 Nov 03 03:38:29.333647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.333227 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerDied","Data":"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70"} Nov 03 03:38:29.333725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.333708 5275 scope.go:122] "RemoveContainer" containerID="32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70" Nov 03 03:38:29.335462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.335433 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:38:29.335559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.335503 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:38:29.335559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:29.335537 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:38:30.342184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:30.342151 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerStarted","Data":"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f"} Nov 03 03:38:31.264307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:31.264272 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:38:31.264683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:31.264660 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:38:31.264831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:31.264807 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:38:32.263874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:32.262318 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:38:32.263874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:32.262386 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:38:32.331122 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:38:32.331090 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:38:33.265873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:33.265281 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:38:33.265873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:33.265344 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:38:34.270364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:34.270323 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:38:34.270978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:34.270942 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:38:35.261797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:35.261266 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:38:35.261797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:35.261370 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:38:35.303123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:35.303090 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:38:36.266004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:36.265974 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:38:36.266453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:36.266434 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:38:36.267219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:36.267196 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:38:36.309037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:36.308998 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:38:51.909832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:51.909783 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:38:56.910798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:56.910748 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" podUID="0400fc12-9eb6-4cc6-be94-cb19130877b6" containerName="payload-pre-processing" containerID="cri-o://2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903" gracePeriod=25 Nov 03 03:38:57.179984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.179920 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:38:57.237801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.237760 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\" (UniqueName: \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\") pod \"0400fc12-9eb6-4cc6-be94-cb19130877b6\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " Nov 03 03:38:57.238028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.237833 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\") pod \"0400fc12-9eb6-4cc6-be94-cb19130877b6\" (UID: \"0400fc12-9eb6-4cc6-be94-cb19130877b6\") " Nov 03 03:38:57.238217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.238193 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume" pod "0400fc12-9eb6-4cc6-be94-cb19130877b6" (UID: "0400fc12-9eb6-4cc6-be94-cb19130877b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:38:57.238487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.238469 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400fc12-9eb6-4cc6-be94-cb19130877b6-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:57.240134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.240109 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l" pod "0400fc12-9eb6-4cc6-be94-cb19130877b6" (UID: "0400fc12-9eb6-4cc6-be94-cb19130877b6"). InnerVolumeSpecName "kube-api-access-9lq8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:38:57.339509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.339476 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lq8l\" (UniqueName: \"kubernetes.io/projected/0400fc12-9eb6-4cc6-be94-cb19130877b6-kube-api-access-9lq8l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:57.649586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.649546 5275 generic.go:356] "Generic (PLEG): container finished" podID="0400fc12-9eb6-4cc6-be94-cb19130877b6" containerID="2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903" exitCode=0 Nov 03 03:38:57.649761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.649600 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" Nov 03 03:38:57.649761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.649628 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" event={"ID":"0400fc12-9eb6-4cc6-be94-cb19130877b6","Type":"ContainerDied","Data":"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903"} Nov 03 03:38:57.649761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.649699 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r" event={"ID":"0400fc12-9eb6-4cc6-be94-cb19130877b6","Type":"ContainerDied","Data":"255a43eedf201be4c38f30bfb4d7de1cdbde3ba5066145523acb4725ba8e7be3"} Nov 03 03:38:57.649761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.649724 5275 scope.go:122] "RemoveContainer" containerID="2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903" Nov 03 03:38:57.660149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.660130 5275 scope.go:122] "RemoveContainer" containerID="2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903" Nov 03 03:38:57.660467 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:38:57.660438 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903\": container with ID starting with 2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903 not found: ID does not exist" containerID="2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903" Nov 03 03:38:57.660526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.660477 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903"} err="failed to get container status \"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903\": rpc error: code = NotFound desc = could not find container \"2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903\": container with ID starting with 2d856fcaf3e017e9980b637b235c88908246abffc0f09e932a886e1a8c8b1903 not found: ID does not exist" Nov 03 03:38:57.816892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.816841 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:38:57.867709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:57.867672 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-6f4c846687-2nd4r"] Nov 03 03:38:58.987223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:38:58.987184 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0400fc12-9eb6-4cc6-be94-cb19130877b6" path="/var/lib/kubelet/pods/0400fc12-9eb6-4cc6-be94-cb19130877b6/volumes" Nov 03 03:39:02.329828 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:02.329787 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:39:29.526542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:29.526458 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:39:29.527175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:29.526743 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" containerID="cri-o://27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1" gracePeriod=30 Nov 03 03:39:31.977107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:31.977059 5275 patch_prober.go:29] interesting pod/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:39:31.977658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:31.977138 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:39:32.330816 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:32.330781 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:39:34.815767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.815743 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:39:34.886364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886330 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886378 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886457 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886506 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886569 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886610 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886658 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.886733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886690 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:34.887048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886920 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.887048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.887040 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\") pod \"e7069423-18e4-4e44-84de-76ae9bd70f38\" (UID: \"e7069423-18e4-4e44-84de-76ae9bd70f38\") " Nov 03 03:39:34.887433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.886935 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:34.887571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.887367 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:34.887662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.887637 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.887722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.887670 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.888030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.887986 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:34.888391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.888370 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:34.889099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.889071 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:34.889689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.889655 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:39:34.889954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.889932 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:34.890598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.890572 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7" pod "e7069423-18e4-4e44-84de-76ae9bd70f38" (UID: "e7069423-18e4-4e44-84de-76ae9bd70f38"). InnerVolumeSpecName "kube-api-access-jt7c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:34.960130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.960096 5275 generic.go:356] "Generic (PLEG): container finished" podID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerID="27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1" exitCode=0 Nov 03 03:39:34.960373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.960182 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" Nov 03 03:39:34.960373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.960181 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerDied","Data":"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1"} Nov 03 03:39:34.960373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.960281 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89" event={"ID":"e7069423-18e4-4e44-84de-76ae9bd70f38","Type":"ContainerDied","Data":"8327f4bfd10d60a01bcd2217bc3b2b4f0e639eadbd339d1fcadb97aaebfd6e5b"} Nov 03 03:39:34.960373 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.960311 5275 scope.go:122] "RemoveContainer" containerID="27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1" Nov 03 03:39:34.971622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.971512 5275 scope.go:122] "RemoveContainer" containerID="fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db" Nov 03 03:39:34.982985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.982959 5275 scope.go:122] "RemoveContainer" containerID="27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1" Nov 03 03:39:34.983273 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:34.983252 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1\": container with ID starting with 27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1 not found: ID does not exist" containerID="27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1" Nov 03 03:39:34.983387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.983281 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1"} err="failed to get container status \"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1\": rpc error: code = NotFound desc = could not find container \"27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1\": container with ID starting with 27aa189a6fb395df8d03ffd36fddb033612db9536db7c8aea3759335af7609f1 not found: ID does not exist" Nov 03 03:39:34.983387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.983302 5275 scope.go:122] "RemoveContainer" containerID="fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db" Nov 03 03:39:34.983517 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:34.983507 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db\": container with ID starting with fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db not found: ID does not exist" containerID="fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db" Nov 03 03:39:34.983577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.983525 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db"} err="failed to get container status \"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db\": rpc error: code = NotFound desc = could not find container \"fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db\": container with ID starting with fb63c6a98fc0450831c2ba4c29f0214b91540805ca73cb3ec67ad129a7ef35db not found: ID does not exist" Nov 03 03:39:34.989537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989506 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989540 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989554 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989569 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069423-18e4-4e44-84de-76ae9bd70f38-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989583 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7c7\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-kube-api-access-jt7c7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989596 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069423-18e4-4e44-84de-76ae9bd70f38-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:34.989769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:34.989608 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069423-18e4-4e44-84de-76ae9bd70f38-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:35.133239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:35.133143 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:39:35.140047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:35.140011 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-95ab58-openshift-default-6c99678b75-2kh89"] Nov 03 03:39:36.985723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:36.985678 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" path="/var/lib/kubelet/pods/e7069423-18e4-4e44-84de-76ae9bd70f38/volumes" Nov 03 03:39:40.925904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:40.925859 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:39:40.926903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:40.926576 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" containerID="cri-o://a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf" gracePeriod=30 Nov 03 03:39:41.035011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:41.034974 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:39:41.035319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:41.035261 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" containerID="cri-o://6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c" gracePeriod=30 Nov 03 03:39:43.227243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:43.227198 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:39:43.227779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:43.227274 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:39:43.337670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:43.337638 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:39:43.337859 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:43.337688 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:39:44.009084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:44.009037 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:39:44.009306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:44.009118 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:39:46.087936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.087891 5275 generic.go:356] "Generic (PLEG): container finished" podID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerID="471e7492ac7a70b8daf3b6abf727aaa26a771be9898c144cd41e283ca0dbeed3" exitCode=0 Nov 03 03:39:46.088763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.088705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerDied","Data":"471e7492ac7a70b8daf3b6abf727aaa26a771be9898c144cd41e283ca0dbeed3"} Nov 03 03:39:46.089309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.089289 5275 scope.go:122] "RemoveContainer" containerID="471e7492ac7a70b8daf3b6abf727aaa26a771be9898c144cd41e283ca0dbeed3" Nov 03 03:39:46.104585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.104522 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:39:46.104762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.104647 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:39:46.104762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.104696 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:39:46.418973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.418946 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:39:46.424612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.424586 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:39:46.503322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503237 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.503322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503280 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.503610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503386 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.503610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503459 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.503610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503592 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.503811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503657 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.503811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503739 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.503811 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503776 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.504034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503879 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.504034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503917 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.504034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503955 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.504034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.503999 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504033 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504070 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504120 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504170 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504233 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\") pod \"7ac85226-99d9-4c5f-838f-3e51581078b5\" (UID: \"7ac85226-99d9-4c5f-838f-3e51581078b5\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504278 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\") pod \"bf073012-438e-48f7-b1b4-d5230919e381\" (UID: \"bf073012-438e-48f7-b1b4-d5230919e381\") " Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504280 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504593 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504612 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.504878 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505282 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505303 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505319 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505334 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac85226-99d9-4c5f-838f-3e51581078b5-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505715 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505868 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.505999 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.506127 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.506308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.506485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.506328 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.508429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.507293 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:46.508429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.507552 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:39:46.508429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.508319 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.508429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.508368 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr" pod "bf073012-438e-48f7-b1b4-d5230919e381" (UID: "bf073012-438e-48f7-b1b4-d5230919e381"). InnerVolumeSpecName "kube-api-access-b8rqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:46.509455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.509431 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:46.509671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.509643 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:39:46.510170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.510138 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:46.511176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.511142 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9" pod "7ac85226-99d9-4c5f-838f-3e51581078b5" (UID: "7ac85226-99d9-4c5f-838f-3e51581078b5"). InnerVolumeSpecName "kube-api-access-zpbp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:46.606571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606527 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf073012-438e-48f7-b1b4-d5230919e381-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606568 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606580 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606591 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606601 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606615 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606625 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf073012-438e-48f7-b1b4-d5230919e381-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606634 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rqr\" (UniqueName: \"kubernetes.io/projected/bf073012-438e-48f7-b1b4-d5230919e381-kube-api-access-b8rqr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606644 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606653 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606662 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac85226-99d9-4c5f-838f-3e51581078b5-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606672 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac85226-99d9-4c5f-838f-3e51581078b5-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606682 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbp9\" (UniqueName: \"kubernetes.io/projected/7ac85226-99d9-4c5f-838f-3e51581078b5-kube-api-access-zpbp9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:46.606824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:46.606691 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf073012-438e-48f7-b1b4-d5230919e381-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:47.112347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.112311 5275 generic.go:356] "Generic (PLEG): container finished" podID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerID="6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c" exitCode=0 Nov 03 03:39:47.112828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.112362 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" Nov 03 03:39:47.112828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.112511 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" event={"ID":"7ac85226-99d9-4c5f-838f-3e51581078b5","Type":"ContainerDied","Data":"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c"} Nov 03 03:39:47.112828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.112555 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9" event={"ID":"7ac85226-99d9-4c5f-838f-3e51581078b5","Type":"ContainerDied","Data":"8e0d9423efc39a47438797bb06be9488658026bc4917c4795baf2cc2462221c3"} Nov 03 03:39:47.112828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.112582 5275 scope.go:122] "RemoveContainer" containerID="6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c" Nov 03 03:39:47.116623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.116213 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerStarted","Data":"6043c036127fbc11d5dfa1a31777e558aa0b30d39b20b0000107552345114605"} Nov 03 03:39:47.118374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.118346 5275 generic.go:356] "Generic (PLEG): container finished" podID="bf073012-438e-48f7-b1b4-d5230919e381" containerID="a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf" exitCode=0 Nov 03 03:39:47.118479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.118399 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" Nov 03 03:39:47.118479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.118422 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" event={"ID":"bf073012-438e-48f7-b1b4-d5230919e381","Type":"ContainerDied","Data":"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf"} Nov 03 03:39:47.118479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.118462 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl" event={"ID":"bf073012-438e-48f7-b1b4-d5230919e381","Type":"ContainerDied","Data":"41d3a3b9643e549ae87bc1b8df0d2efb9f81036970a2dcb7319d5b566b636e37"} Nov 03 03:39:47.130665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.130646 5275 scope.go:122] "RemoveContainer" containerID="6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c" Nov 03 03:39:47.131618 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:47.131589 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c\": container with ID starting with 6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c not found: ID does not exist" containerID="6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c" Nov 03 03:39:47.131774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.131749 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c"} err="failed to get container status \"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c\": rpc error: code = NotFound desc = could not find container \"6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c\": container with ID starting with 6e19bee43fb5018a1daa2846decc16bf0ff82734669c06353b07d47a4764470c not found: ID does not exist" Nov 03 03:39:47.131912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.131899 5275 scope.go:122] "RemoveContainer" containerID="a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf" Nov 03 03:39:47.142400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.142379 5275 scope.go:122] "RemoveContainer" containerID="a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf" Nov 03 03:39:47.142909 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:39:47.142886 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf\": container with ID starting with a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf not found: ID does not exist" containerID="a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf" Nov 03 03:39:47.143078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.143052 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf"} err="failed to get container status \"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf\": rpc error: code = NotFound desc = could not find container \"a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf\": container with ID starting with a73697499030813faabb2099555805a4851a355f603a75971336ae3889f585cf not found: ID does not exist" Nov 03 03:39:47.193396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.193338 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:39:47.193692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.193671 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:47.193773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.193717 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:47.288528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.288494 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:39:47.291536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.291466 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-knkbl"] Nov 03 03:39:47.524102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.524062 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:39:47.528163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:47.528132 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-g79l9"] Nov 03 03:39:48.193637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:48.193598 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:48.194227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:48.193667 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:48.986328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:48.986294 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" path="/var/lib/kubelet/pods/7ac85226-99d9-4c5f-838f-3e51581078b5/volumes" Nov 03 03:39:48.986903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:48.986881 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf073012-438e-48f7-b1b4-d5230919e381" path="/var/lib/kubelet/pods/bf073012-438e-48f7-b1b4-d5230919e381/volumes" Nov 03 03:39:49.193200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:49.193167 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:49.193350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:49.193224 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:50.193938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:50.193906 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:50.194338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:50.193964 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:51.193553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:51.193510 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:51.193764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:51.193572 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:52.193440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:52.193400 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 03:39:52.193866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:52.193485 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 03:39:53.199018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:53.198985 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:39:53.199463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:53.199394 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:39:53.200321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:39:53.200303 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:40:02.329531 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:40:02.329497 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:40:03.967776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:03.967735 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:40:03.968314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:03.968053 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" podUID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" containerName="payload-processing" containerID="cri-o://71f4c57ac2a4bcdd415cceece57fb8a3498a1f169993ebc08f8766e866810e5e" gracePeriod=30 Nov 03 03:40:04.023810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.023773 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:40:04.024079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.024052 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" podUID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" containerName="payload-pre-processing" containerID="cri-o://28c08fe7ed15a1f9bd353ed730e511cb8991dfae9661e76fc61bfa7e5e058536" gracePeriod=30 Nov 03 03:40:04.267082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.267043 5275 generic.go:356] "Generic (PLEG): container finished" podID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" containerID="71f4c57ac2a4bcdd415cceece57fb8a3498a1f169993ebc08f8766e866810e5e" exitCode=0 Nov 03 03:40:04.267276 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.267161 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" event={"ID":"d9f0341e-c8f4-4941-bf8c-8721bcd1277b","Type":"ContainerDied","Data":"71f4c57ac2a4bcdd415cceece57fb8a3498a1f169993ebc08f8766e866810e5e"} Nov 03 03:40:04.308525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.308476 5275 generic.go:356] "Generic (PLEG): container finished" podID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" containerID="28c08fe7ed15a1f9bd353ed730e511cb8991dfae9661e76fc61bfa7e5e058536" exitCode=0 Nov 03 03:40:04.308946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.308765 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" event={"ID":"25b57a00-14c8-4fd1-bf47-cadbabe91a26","Type":"ContainerDied","Data":"28c08fe7ed15a1f9bd353ed730e511cb8991dfae9661e76fc61bfa7e5e058536"} Nov 03 03:40:04.415198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.413149 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:40:04.447733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.447707 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:40:04.521126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521104 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\" (UniqueName: \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\") pod \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " Nov 03 03:40:04.521282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521231 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\") pod \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\" (UID: \"25b57a00-14c8-4fd1-bf47-cadbabe91a26\") " Nov 03 03:40:04.521325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521301 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\" (UniqueName: \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\") pod \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " Nov 03 03:40:04.521384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521340 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\") pod \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\" (UID: \"d9f0341e-c8f4-4941-bf8c-8721bcd1277b\") " Nov 03 03:40:04.521663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521636 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume" pod "25b57a00-14c8-4fd1-bf47-cadbabe91a26" (UID: "25b57a00-14c8-4fd1-bf47-cadbabe91a26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:04.521989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521709 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume" pod "d9f0341e-c8f4-4941-bf8c-8721bcd1277b" (UID: "d9f0341e-c8f4-4941-bf8c-8721bcd1277b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:04.521989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521937 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b57a00-14c8-4fd1-bf47-cadbabe91a26-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:04.521989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.521953 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:04.523544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.523519 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h" pod "25b57a00-14c8-4fd1-bf47-cadbabe91a26" (UID: "25b57a00-14c8-4fd1-bf47-cadbabe91a26"). InnerVolumeSpecName "kube-api-access-7vp5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:04.523640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.523551 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6" pod "d9f0341e-c8f4-4941-bf8c-8721bcd1277b" (UID: "d9f0341e-c8f4-4941-bf8c-8721bcd1277b"). InnerVolumeSpecName "kube-api-access-8cfk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:04.627873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.627438 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfk6\" (UniqueName: \"kubernetes.io/projected/d9f0341e-c8f4-4941-bf8c-8721bcd1277b-kube-api-access-8cfk6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:04.627873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:04.627478 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vp5h\" (UniqueName: \"kubernetes.io/projected/25b57a00-14c8-4fd1-bf47-cadbabe91a26-kube-api-access-7vp5h\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:05.317701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.317668 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" Nov 03 03:40:05.318184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.317669 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx" event={"ID":"d9f0341e-c8f4-4941-bf8c-8721bcd1277b","Type":"ContainerDied","Data":"cdff35d215f09dadcd5bbcda1f36dcdfdffb93b52e4b103cccf2b4d3fe03ee1f"} Nov 03 03:40:05.318184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.317819 5275 scope.go:122] "RemoveContainer" containerID="71f4c57ac2a4bcdd415cceece57fb8a3498a1f169993ebc08f8766e866810e5e" Nov 03 03:40:05.319702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.319680 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" Nov 03 03:40:05.319833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.319705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw" event={"ID":"25b57a00-14c8-4fd1-bf47-cadbabe91a26","Type":"ContainerDied","Data":"2ee525717c860af656c346b88d7db874e8aaf35fc79e56026a987834eefa0006"} Nov 03 03:40:05.329181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.329162 5275 scope.go:122] "RemoveContainer" containerID="28c08fe7ed15a1f9bd353ed730e511cb8991dfae9661e76fc61bfa7e5e058536" Nov 03 03:40:05.487007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.486969 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:40:05.537013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.536978 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-656b74948-742fx"] Nov 03 03:40:05.610537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.610454 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:40:05.613921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:05.613893 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-79cdc489c5-rmhgw"] Nov 03 03:40:06.985598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:06.985561 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" path="/var/lib/kubelet/pods/25b57a00-14c8-4fd1-bf47-cadbabe91a26/volumes" Nov 03 03:40:06.986038 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:06.986010 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" path="/var/lib/kubelet/pods/d9f0341e-c8f4-4941-bf8c-8721bcd1277b/volumes" Nov 03 03:40:32.331176 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:40:32.331134 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:40:32.581441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:32.581326 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:40:32.694682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:32.694641 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:40:37.582431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.582385 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" podUID="945ce218-284d-4d5d-94df-eb45e0ccb44b" containerName="payload-processing" containerID="cri-o://f1b467a0b964f5f5a46edefecf423cbad62ecad48c2a6e53a487ebb229b47f10" gracePeriod=25 Nov 03 03:40:37.695590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.695547 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" podUID="f9ece53d-af9a-4022-acd6-10084d8aed6b" containerName="payload-pre-processing" containerID="cri-o://9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900" gracePeriod=25 Nov 03 03:40:37.731061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.731015 5275 generic.go:356] "Generic (PLEG): container finished" podID="945ce218-284d-4d5d-94df-eb45e0ccb44b" containerID="f1b467a0b964f5f5a46edefecf423cbad62ecad48c2a6e53a487ebb229b47f10" exitCode=0 Nov 03 03:40:37.731227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.731073 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" event={"ID":"945ce218-284d-4d5d-94df-eb45e0ccb44b","Type":"ContainerDied","Data":"f1b467a0b964f5f5a46edefecf423cbad62ecad48c2a6e53a487ebb229b47f10"} Nov 03 03:40:37.860864 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.860828 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:40:37.974186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:37.974159 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:40:38.035516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.035481 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\" (UniqueName: \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\") pod \"945ce218-284d-4d5d-94df-eb45e0ccb44b\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " Nov 03 03:40:38.035722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.035628 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\") pod \"945ce218-284d-4d5d-94df-eb45e0ccb44b\" (UID: \"945ce218-284d-4d5d-94df-eb45e0ccb44b\") " Nov 03 03:40:38.036111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.036082 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume" pod "945ce218-284d-4d5d-94df-eb45e0ccb44b" (UID: "945ce218-284d-4d5d-94df-eb45e0ccb44b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:38.036312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.036294 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945ce218-284d-4d5d-94df-eb45e0ccb44b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:38.037789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.037766 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975" pod "945ce218-284d-4d5d-94df-eb45e0ccb44b" (UID: "945ce218-284d-4d5d-94df-eb45e0ccb44b"). InnerVolumeSpecName "kube-api-access-8d975". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:38.136975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.136893 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\") pod \"f9ece53d-af9a-4022-acd6-10084d8aed6b\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " Nov 03 03:40:38.136975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.136961 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\" (UniqueName: \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\") pod \"f9ece53d-af9a-4022-acd6-10084d8aed6b\" (UID: \"f9ece53d-af9a-4022-acd6-10084d8aed6b\") " Nov 03 03:40:38.137287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.137258 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume" pod "f9ece53d-af9a-4022-acd6-10084d8aed6b" (UID: "f9ece53d-af9a-4022-acd6-10084d8aed6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:38.137533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.137518 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d975\" (UniqueName: \"kubernetes.io/projected/945ce218-284d-4d5d-94df-eb45e0ccb44b-kube-api-access-8d975\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:38.137580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.137535 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ece53d-af9a-4022-acd6-10084d8aed6b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:38.139329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.139303 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2" pod "f9ece53d-af9a-4022-acd6-10084d8aed6b" (UID: "f9ece53d-af9a-4022-acd6-10084d8aed6b"). InnerVolumeSpecName "kube-api-access-sn6x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:38.239182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.239142 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6x2\" (UniqueName: \"kubernetes.io/projected/f9ece53d-af9a-4022-acd6-10084d8aed6b-kube-api-access-sn6x2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:38.738968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.738930 5275 generic.go:356] "Generic (PLEG): container finished" podID="f9ece53d-af9a-4022-acd6-10084d8aed6b" containerID="9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900" exitCode=0 Nov 03 03:40:38.739446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.738973 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" Nov 03 03:40:38.739446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.739019 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" event={"ID":"f9ece53d-af9a-4022-acd6-10084d8aed6b","Type":"ContainerDied","Data":"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900"} Nov 03 03:40:38.739446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.739072 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd" event={"ID":"f9ece53d-af9a-4022-acd6-10084d8aed6b","Type":"ContainerDied","Data":"adbebbedb2df2e1570cb0f6d2a5a967a8eb49dea4083c928e588214ae51c0eda"} Nov 03 03:40:38.739446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.739101 5275 scope.go:122] "RemoveContainer" containerID="9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900" Nov 03 03:40:38.740786 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.740759 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" event={"ID":"945ce218-284d-4d5d-94df-eb45e0ccb44b","Type":"ContainerDied","Data":"3838f9398ad839273018692c08780f53bc1a2fd8bc30b72beaace7834e44e84d"} Nov 03 03:40:38.740951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.740896 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v" Nov 03 03:40:38.752601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.752579 5275 scope.go:122] "RemoveContainer" containerID="9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900" Nov 03 03:40:38.752951 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:40:38.752920 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900\": container with ID starting with 9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900 not found: ID does not exist" containerID="9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900" Nov 03 03:40:38.753065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.752962 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900"} err="failed to get container status \"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900\": rpc error: code = NotFound desc = could not find container \"9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900\": container with ID starting with 9e6b27acc1773e4bec248b840e815a8d4de2e10df03422f55694bda21688c900 not found: ID does not exist" Nov 03 03:40:38.753065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.752987 5275 scope.go:122] "RemoveContainer" containerID="f1b467a0b964f5f5a46edefecf423cbad62ecad48c2a6e53a487ebb229b47f10" Nov 03 03:40:38.904151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.904047 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:40:38.907511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.907484 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-77c5f6d9f-gwl8v"] Nov 03 03:40:38.985652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:38.985612 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945ce218-284d-4d5d-94df-eb45e0ccb44b" path="/var/lib/kubelet/pods/945ce218-284d-4d5d-94df-eb45e0ccb44b/volumes" Nov 03 03:40:39.016321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:39.016287 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:40:39.022703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:39.022671 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-77584968cf-lhlnd"] Nov 03 03:40:40.771812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:40.771771 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:40:40.772220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:40.772063 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" podUID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" containerName="payload-processing" containerID="cri-o://f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d" gracePeriod=30 Nov 03 03:40:40.880044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:40.880006 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:40:40.880570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:40.880536 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" podUID="b64fd76f-5f62-4994-87cd-3389d6b3319f" containerName="payload-pre-processing" containerID="cri-o://c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476" gracePeriod=30 Nov 03 03:40:40.991006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:40.990969 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ece53d-af9a-4022-acd6-10084d8aed6b" path="/var/lib/kubelet/pods/f9ece53d-af9a-4022-acd6-10084d8aed6b/volumes" Nov 03 03:40:41.096549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.096240 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:40:41.175197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.175166 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\" (UniqueName: \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\") pod \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " Nov 03 03:40:41.175363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.175237 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\") pod \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\" (UID: \"e32d82b3-c0e1-4298-bdb1-e58b37af8639\") " Nov 03 03:40:41.175744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.175718 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume" pod "e32d82b3-c0e1-4298-bdb1-e58b37af8639" (UID: "e32d82b3-c0e1-4298-bdb1-e58b37af8639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:41.177681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.177660 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j" pod "e32d82b3-c0e1-4298-bdb1-e58b37af8639" (UID: "e32d82b3-c0e1-4298-bdb1-e58b37af8639"). InnerVolumeSpecName "kube-api-access-s9d8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:41.179433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.179418 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:40:41.276901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.276489 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\" (UniqueName: \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\") pod \"b64fd76f-5f62-4994-87cd-3389d6b3319f\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " Nov 03 03:40:41.276901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.276617 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\") pod \"b64fd76f-5f62-4994-87cd-3389d6b3319f\" (UID: \"b64fd76f-5f62-4994-87cd-3389d6b3319f\") " Nov 03 03:40:41.277345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.277042 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9d8j\" (UniqueName: \"kubernetes.io/projected/e32d82b3-c0e1-4298-bdb1-e58b37af8639-kube-api-access-s9d8j\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:41.277345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.277064 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32d82b3-c0e1-4298-bdb1-e58b37af8639-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:41.311928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.277762 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume" pod "b64fd76f-5f62-4994-87cd-3389d6b3319f" (UID: "b64fd76f-5f62-4994-87cd-3389d6b3319f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:41.391904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.387619 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64fd76f-5f62-4994-87cd-3389d6b3319f-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:41.391904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.389530 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc" pod "b64fd76f-5f62-4994-87cd-3389d6b3319f" (UID: "b64fd76f-5f62-4994-87cd-3389d6b3319f"). InnerVolumeSpecName "kube-api-access-j58wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:41.489437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.489394 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58wc\" (UniqueName: \"kubernetes.io/projected/b64fd76f-5f62-4994-87cd-3389d6b3319f-kube-api-access-j58wc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:41.768869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.768751 5275 generic.go:356] "Generic (PLEG): container finished" podID="b64fd76f-5f62-4994-87cd-3389d6b3319f" containerID="c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476" exitCode=0 Nov 03 03:40:41.768869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.768819 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" Nov 03 03:40:41.768869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.768840 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" event={"ID":"b64fd76f-5f62-4994-87cd-3389d6b3319f","Type":"ContainerDied","Data":"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476"} Nov 03 03:40:41.769147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.768951 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj" event={"ID":"b64fd76f-5f62-4994-87cd-3389d6b3319f","Type":"ContainerDied","Data":"27640054cec1526ece01643b702d90b6ce9c1ff123b551dd09c754a737f73d33"} Nov 03 03:40:41.769147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.768980 5275 scope.go:122] "RemoveContainer" containerID="c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476" Nov 03 03:40:41.770473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.770446 5275 generic.go:356] "Generic (PLEG): container finished" podID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" containerID="f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d" exitCode=0 Nov 03 03:40:41.770616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.770530 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" Nov 03 03:40:41.770616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.770557 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" event={"ID":"e32d82b3-c0e1-4298-bdb1-e58b37af8639","Type":"ContainerDied","Data":"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d"} Nov 03 03:40:41.770616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.770598 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w" event={"ID":"e32d82b3-c0e1-4298-bdb1-e58b37af8639","Type":"ContainerDied","Data":"dc3a11c33ff25bb4a6be07cacda509c361109f5e7dd404cdc84985de6500447a"} Nov 03 03:40:41.780308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.780141 5275 scope.go:122] "RemoveContainer" containerID="c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476" Nov 03 03:40:41.780601 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:40:41.780485 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476\": container with ID starting with c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476 not found: ID does not exist" containerID="c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476" Nov 03 03:40:41.780601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.780513 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476"} err="failed to get container status \"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476\": rpc error: code = NotFound desc = could not find container \"c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476\": container with ID starting with c643bd1b6133af656b4f1aa5c496269bce5a059fb3391ac9830c3fb573567476 not found: ID does not exist" Nov 03 03:40:41.780601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.780529 5275 scope.go:122] "RemoveContainer" containerID="f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d" Nov 03 03:40:41.802071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.802044 5275 scope.go:122] "RemoveContainer" containerID="f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d" Nov 03 03:40:41.802445 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:40:41.802422 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d\": container with ID starting with f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d not found: ID does not exist" containerID="f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d" Nov 03 03:40:41.802527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.802451 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d"} err="failed to get container status \"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d\": rpc error: code = NotFound desc = could not find container \"f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d\": container with ID starting with f932d2c613d8dcabdb027431ceaeb0745f65d1de6fad8bd383d1d169a31f361d not found: ID does not exist" Nov 03 03:40:41.936232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.936198 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:40:41.940814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:41.940788 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-564f6cbb4d-k9f8w"] Nov 03 03:40:42.047131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:42.047098 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:40:42.050988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:42.050952 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-bd986ddbc-lx6sj"] Nov 03 03:40:42.986613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:42.986581 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64fd76f-5f62-4994-87cd-3389d6b3319f" path="/var/lib/kubelet/pods/b64fd76f-5f62-4994-87cd-3389d6b3319f/volumes" Nov 03 03:40:42.987068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:42.987013 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" path="/var/lib/kubelet/pods/e32d82b3-c0e1-4298-bdb1-e58b37af8639/volumes" Nov 03 03:40:50.774611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.774562 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:40:50.775451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775423 5275 cpu_manager.go:414] "containerMap: removing container" podUID="945ce218-284d-4d5d-94df-eb45e0ccb44b" containerName="payload-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775453 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945ce218-284d-4d5d-94df-eb45e0ccb44b" containerName="payload-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775471 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775479 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775498 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" containerName="payload-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775506 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" containerName="payload-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775521 5275 cpu_manager.go:414] "containerMap: removing container" podUID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775528 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775546 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b64fd76f-5f62-4994-87cd-3389d6b3319f" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775553 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64fd76f-5f62-4994-87cd-3389d6b3319f" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775566 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775573 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775580 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775585 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775598 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0400fc12-9eb6-4cc6-be94-cb19130877b6" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775603 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0400fc12-9eb6-4cc6-be94-cb19130877b6" containerName="payload-pre-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775615 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" containerName="payload-processing" Nov 03 03:40:50.775619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775621 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" containerName="payload-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775632 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775640 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775657 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f9ece53d-af9a-4022-acd6-10084d8aed6b" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775662 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ece53d-af9a-4022-acd6-10084d8aed6b" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775756 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ece53d-af9a-4022-acd6-10084d8aed6b" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775767 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="945ce218-284d-4d5d-94df-eb45e0ccb44b" containerName="payload-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775774 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac85226-99d9-4c5f-838f-3e51581078b5" containerName="istio-proxy" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775781 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32d82b3-c0e1-4298-bdb1-e58b37af8639" containerName="payload-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775788 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b57a00-14c8-4fd1-bf47-cadbabe91a26" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775793 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64fd76f-5f62-4994-87cd-3389d6b3319f" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775798 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775813 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7069423-18e4-4e44-84de-76ae9bd70f38" containerName="istio-proxy" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775824 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0400fc12-9eb6-4cc6-be94-cb19130877b6" containerName="payload-pre-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775833 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f0341e-c8f4-4941-bf8c-8721bcd1277b" containerName="payload-processing" Nov 03 03:40:50.776388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.775839 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf073012-438e-48f7-b1b4-d5230919e381" containerName="istio-proxy" Nov 03 03:40:50.781550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.781526 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.844235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.844195 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b755ee-openshift-default-dockercfg-67t8p\"" Nov 03 03:40:50.903594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.903562 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.942865 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.942967 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943077 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943130 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943211 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8d9\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943262 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943313 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:50.948303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:50.943426 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.047900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.047754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.047920 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.047957 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.047982 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048040 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048082 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048120 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8d9\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048167 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048215 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048260 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048273 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048452 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048490 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.048841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.048820 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.050594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.050571 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.050742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.050724 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.112289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.112249 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.112484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.112365 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8d9\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\") pod \"e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.201646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.201612 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:51.639665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.639638 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:40:51.640568 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:40:51.640544 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192a40d2_037f_4757_919b_dc11688e9b19.slice/crio-4b9d53c4d6aa6b1ef31a5df0798713069b81e1ff8218c14963c05e0ee86d9b38 WatchSource:0}: Error finding container 4b9d53c4d6aa6b1ef31a5df0798713069b81e1ff8218c14963c05e0ee86d9b38: Status 404 returned error can't find the container with id 4b9d53c4d6aa6b1ef31a5df0798713069b81e1ff8218c14963c05e0ee86d9b38 Nov 03 03:40:51.643140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.643108 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:40:51.643210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.643185 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:40:51.643251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.643215 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:40:51.916560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.916469 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerStarted","Data":"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d"} Nov 03 03:40:51.916560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:51.916533 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerStarted","Data":"4b9d53c4d6aa6b1ef31a5df0798713069b81e1ff8218c14963c05e0ee86d9b38"} Nov 03 03:40:52.039171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:52.036957 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podStartSLOduration=2.036937824 podStartE2EDuration="2.036937824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:52.035971091 +0000 UTC m=+2343.188659511" watchObservedRunningTime="2026-11-03 03:40:52.036937824 +0000 UTC m=+2343.189626246" Nov 03 03:40:52.201966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:52.201836 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:52.207418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:52.207393 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:52.923912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:52.923879 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:40:52.924828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:40:52.924809 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:41:02.330251 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:41:02.330214 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:41:31.190055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:31.190015 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:41:31.190610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:31.190339 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" containerID="cri-o://6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f" gracePeriod=30 Nov 03 03:41:32.331373 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:41:32.331336 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:41:34.282570 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.282525 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:41:34.287078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.287054 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.379810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.379765 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.380027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.379960 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bc4b\" (UniqueName: \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.389750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.389710 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b9c10a\"" Nov 03 03:41:34.389989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.389924 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b9c10a-dockercfg-2zsb4\"" Nov 03 03:41:34.392953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.392929 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:41:34.489159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.485360 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.489159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.485459 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bc4b\" (UniqueName: \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.528230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.516123 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.579553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.579465 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bc4b\" (UniqueName: \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\") pod \"payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.608014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.607980 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:34.994365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:34.994338 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:41:34.995502 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:41:34.995477 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09fb47d_651c_4555_b026_1425f65b4148.slice/crio-e3d5d554ca093439067f3c8cff34d3bdff52ec0fd9afd06569a576f5b6600f90 WatchSource:0}: Error finding container e3d5d554ca093439067f3c8cff34d3bdff52ec0fd9afd06569a576f5b6600f90: Status 404 returned error can't find the container with id e3d5d554ca093439067f3c8cff34d3bdff52ec0fd9afd06569a576f5b6600f90 Nov 03 03:41:35.066885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.066827 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:41:35.067101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.067075 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" containerID="cri-o://f33c773509d37523543a81c6b3bd33f36f297e68f90ee972f84321d30e72c561" gracePeriod=30 Nov 03 03:41:35.232659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.232614 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" event={"ID":"a09fb47d-651c-4555-b026-1425f65b4148","Type":"ContainerStarted","Data":"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9"} Nov 03 03:41:35.232882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.232704 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:35.232882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.232732 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" event={"ID":"a09fb47d-651c-4555-b026-1425f65b4148","Type":"ContainerStarted","Data":"e3d5d554ca093439067f3c8cff34d3bdff52ec0fd9afd06569a576f5b6600f90"} Nov 03 03:41:35.352635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:35.352579 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" podStartSLOduration=1.352564186 podStartE2EDuration="1.352564186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:35.34845099 +0000 UTC m=+2386.501139412" watchObservedRunningTime="2026-11-03 03:41:35.352564186 +0000 UTC m=+2386.505252630" Nov 03 03:41:36.268079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.268049 5275 patch_prober.go:29] interesting pod/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 03:41:36.268344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.268103 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 03:41:36.464921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.464828 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:41:36.512953 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.512918 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.512998 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513048 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513085 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513138 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513177 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513270 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513327 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:36.513958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513617 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:36.513958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513895 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.513958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513952 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\") pod \"661b866e-2e3b-4410-ac22-118ad3ea37f3\" (UID: \"661b866e-2e3b-4410-ac22-118ad3ea37f3\") " Nov 03 03:41:36.514487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513657 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:36.514487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.513912 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:36.514487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514305 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:36.514659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514556 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.514659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514576 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.514659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514591 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.514659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514605 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.514659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.514618 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b866e-2e3b-4410-ac22-118ad3ea37f3-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.515417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.515389 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:36.516005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.515982 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:41:36.516685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.516644 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "kube-api-access-7zlds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:36.516799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.516681 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token" pod "661b866e-2e3b-4410-ac22-118ad3ea37f3" (UID: "661b866e-2e3b-4410-ac22-118ad3ea37f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:36.616032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.615997 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zlds\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-kube-api-access-7zlds\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.616032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.616025 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.616032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.616039 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:36.616275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:36.616047 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b866e-2e3b-4410-ac22-118ad3ea37f3-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:37.249793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.249702 5275 generic.go:356] "Generic (PLEG): container finished" podID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerID="6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f" exitCode=0 Nov 03 03:41:37.249793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.249767 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" Nov 03 03:41:37.250040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.249791 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerDied","Data":"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f"} Nov 03 03:41:37.250040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.249869 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs" event={"ID":"661b866e-2e3b-4410-ac22-118ad3ea37f3","Type":"ContainerDied","Data":"ef4dff109e272d7a22e163b0b5e568a1bac78a65514ec7451b8cdafc6496a296"} Nov 03 03:41:37.250040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.249901 5275 scope.go:122] "RemoveContainer" containerID="6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f" Nov 03 03:41:37.261105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.261083 5275 scope.go:122] "RemoveContainer" containerID="32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70" Nov 03 03:41:37.270682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.270659 5275 scope.go:122] "RemoveContainer" containerID="6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f" Nov 03 03:41:37.270999 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:41:37.270978 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f\": container with ID starting with 6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f not found: ID does not exist" containerID="6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f" Nov 03 03:41:37.271049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.271010 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f"} err="failed to get container status \"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f\": rpc error: code = NotFound desc = could not find container \"6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f\": container with ID starting with 6e5bdaf6f82d3c7fc01a58ff5236acae1a190c62addaa4f972d3d00e4195080f not found: ID does not exist" Nov 03 03:41:37.271049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.271031 5275 scope.go:122] "RemoveContainer" containerID="32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70" Nov 03 03:41:37.271317 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:41:37.271297 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70\": container with ID starting with 32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70 not found: ID does not exist" containerID="32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70" Nov 03 03:41:37.271386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.271325 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70"} err="failed to get container status \"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70\": rpc error: code = NotFound desc = could not find container \"32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70\": container with ID starting with 32d442a893c47b8516dac2c8b6289f0219f6bd84cbd45e1edee3fd2d264f3f70 not found: ID does not exist" Nov 03 03:41:37.463907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.463874 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:41:37.513279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:37.513199 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8200d4-openshift-default-867444cdfc-k2brs"] Nov 03 03:41:38.985149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:38.985107 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" path="/var/lib/kubelet/pods/661b866e-2e3b-4410-ac22-118ad3ea37f3/volumes" Nov 03 03:41:40.286547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.284818 5275 generic.go:356] "Generic (PLEG): container finished" podID="944539d9-6813-4673-919b-e896606acb5e" containerID="f33c773509d37523543a81c6b3bd33f36f297e68f90ee972f84321d30e72c561" exitCode=0 Nov 03 03:41:40.286547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.284897 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" event={"ID":"944539d9-6813-4673-919b-e896606acb5e","Type":"ContainerDied","Data":"f33c773509d37523543a81c6b3bd33f36f297e68f90ee972f84321d30e72c561"} Nov 03 03:41:40.361551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.361524 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:41:40.460920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.460837 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.460937 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461018 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461073 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461100 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461151 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461186 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461215 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461301 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:40.461341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461322 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\") pod \"944539d9-6813-4673-919b-e896606acb5e\" (UID: \"944539d9-6813-4673-919b-e896606acb5e\") " Nov 03 03:41:40.461593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461409 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:40.461653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461628 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:40.461653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461639 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:40.461880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.461825 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:40.462238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.462216 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.462291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.462246 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.462291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.462261 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944539d9-6813-4673-919b-e896606acb5e-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.462291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.462274 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.462291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.462287 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.463532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.463505 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:40.463656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.463605 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:40.463878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.463819 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:41:40.465015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.464991 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx" pod "944539d9-6813-4673-919b-e896606acb5e" (UID: "944539d9-6813-4673-919b-e896606acb5e"). InnerVolumeSpecName "kube-api-access-nklmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:40.564028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.563989 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944539d9-6813-4673-919b-e896606acb5e-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.564028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.564026 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nklmx\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-kube-api-access-nklmx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.564239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.564040 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944539d9-6813-4673-919b-e896606acb5e-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:40.564239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:40.564052 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944539d9-6813-4673-919b-e896606acb5e-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:41.294454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:41.294427 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" Nov 03 03:41:41.294973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:41.294425 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49" event={"ID":"944539d9-6813-4673-919b-e896606acb5e","Type":"ContainerDied","Data":"bfe2eb93180dc4f958ed6366805557a92bf228806fbf2c93a51c98a6ca650812"} Nov 03 03:41:41.294973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:41.294571 5275 scope.go:122] "RemoveContainer" containerID="f33c773509d37523543a81c6b3bd33f36f297e68f90ee972f84321d30e72c561" Nov 03 03:41:41.512441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:41.512405 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:41:41.516768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:41.516738 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bvz49"] Nov 03 03:41:42.398892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.398841 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:41:42.399564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399544 5275 cpu_manager.go:414] "containerMap: removing container" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.399615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399568 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.399615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399595 5275 cpu_manager.go:414] "containerMap: removing container" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" Nov 03 03:41:42.399615 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399603 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" Nov 03 03:41:42.399755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399742 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.399810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399760 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="944539d9-6813-4673-919b-e896606acb5e" containerName="istio-proxy" Nov 03 03:41:42.399810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.399770 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.404929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.404896 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.465877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.464952 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-29dd3f\"" Nov 03 03:41:42.472938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.472883 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:41:42.534893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.524422 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.535260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.535171 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dmzh\" (UniqueName: \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.535719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.528434 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-29dd3f-dockercfg-qhmm2\"" Nov 03 03:41:42.566136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.564805 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:41:42.566136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.565759 5275 cpu_manager.go:414] "containerMap: removing container" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.566136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.565781 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661b866e-2e3b-4410-ac22-118ad3ea37f3" containerName="istio-proxy" Nov 03 03:41:42.570113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.569976 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.595928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.595893 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:41:42.637075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.637045 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.637277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.637095 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dmzh\" (UniqueName: \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.637732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.637712 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.698015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.697940 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dmzh\" (UniqueName: \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\") pod \"payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.721515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.721483 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:42.738383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.738348 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdznb\" (UniqueName: \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.738602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.738552 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.840155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.840121 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdznb\" (UniqueName: \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.840320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.840240 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.841164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.841127 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.900761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.900731 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdznb\" (UniqueName: \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\") pod \"payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:42.986872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:42.986775 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944539d9-6813-4673-919b-e896606acb5e" path="/var/lib/kubelet/pods/944539d9-6813-4673-919b-e896606acb5e/volumes" Nov 03 03:41:43.107307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.107274 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:41:43.108548 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:41:43.108523 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b823af_7342_4b02_8781_63b5031824f4.slice/crio-7831ac39c37202faba0127c8ae9a56e5c322a254908b132087ab5380b70f5d78 WatchSource:0}: Error finding container 7831ac39c37202faba0127c8ae9a56e5c322a254908b132087ab5380b70f5d78: Status 404 returned error can't find the container with id 7831ac39c37202faba0127c8ae9a56e5c322a254908b132087ab5380b70f5d78 Nov 03 03:41:43.188704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.188672 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:43.325682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.325637 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" event={"ID":"46b823af-7342-4b02-8781-63b5031824f4","Type":"ContainerStarted","Data":"4f5b13a2840a7294338be1a0aba63f3bcf2ebf51f8be541e0c50168806fd9b36"} Nov 03 03:41:43.325682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.325688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" event={"ID":"46b823af-7342-4b02-8781-63b5031824f4","Type":"ContainerStarted","Data":"7831ac39c37202faba0127c8ae9a56e5c322a254908b132087ab5380b70f5d78"} Nov 03 03:41:43.325986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.325753 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:43.442571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.442211 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" podStartSLOduration=1.4421932929999999 podStartE2EDuration="1.442193293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:43.44109663 +0000 UTC m=+2394.593785051" watchObservedRunningTime="2026-11-03 03:41:43.442193293 +0000 UTC m=+2394.594881717" Nov 03 03:41:43.572960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:43.572693 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:41:43.579504 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:41:43.578710 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1e5522_7ac0_4368_b248_1d8a0b736147.slice/crio-1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56 WatchSource:0}: Error finding container 1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56: Status 404 returned error can't find the container with id 1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56 Nov 03 03:41:44.333352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:44.333311 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" event={"ID":"3c1e5522-7ac0-4368-b248-1d8a0b736147","Type":"ContainerStarted","Data":"3877d5aee90fc29b6d3f3172dbe0eedafc250490475a0e4322db75bc6f5032fd"} Nov 03 03:41:44.333539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:44.333371 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" event={"ID":"3c1e5522-7ac0-4368-b248-1d8a0b736147","Type":"ContainerStarted","Data":"1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56"} Nov 03 03:41:44.333601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:44.333568 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:44.450809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:44.450748 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" podStartSLOduration=2.450729575 podStartE2EDuration="2.450729575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:44.448411722 +0000 UTC m=+2395.601100144" watchObservedRunningTime="2026-11-03 03:41:44.450729575 +0000 UTC m=+2395.603417999" Nov 03 03:41:46.242235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:46.242205 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:41:49.601906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:49.601875 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:41:49.608396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:49.608375 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:41:52.925426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:52.925379 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:41:52.925946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:52.925643 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" podUID="cbd59738-5074-4422-913f-1d9e55ead178" containerName="discovery" containerID="cri-o://cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e" gracePeriod=30 Nov 03 03:41:53.202703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.202677 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:41:53.289085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289046 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.289300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289145 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.289300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289239 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.289411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289310 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.289585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289561 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.289767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.289609 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.290496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.290467 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\") pod \"cbd59738-5074-4422-913f-1d9e55ead178\" (UID: \"cbd59738-5074-4422-913f-1d9e55ead178\") " Nov 03 03:41:53.291084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.291051 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:53.291238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.291203 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-ca-configmap\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.292160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.292130 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:53.292621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.292590 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:53.293653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.293620 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:53.293757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.293652 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:53.293986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.293959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "kube-api-access-97bcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:53.294252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.294232 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert" pod "cbd59738-5074-4422-913f-1d9e55ead178" (UID: "cbd59738-5074-4422-913f-1d9e55ead178"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:53.393302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393260 5275 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-cacerts\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.393302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393299 5275 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd59738-5074-4422-913f-1d9e55ead178-local-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.393571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393315 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bcx\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-kube-api-access-97bcx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.393571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393331 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd59738-5074-4422-913f-1d9e55ead178-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.393571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393345 5275 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-kubeconfig\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.393571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.393359 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbd59738-5074-4422-913f-1d9e55ead178-istio-csr-dns-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:53.400577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.400545 5275 generic.go:356] "Generic (PLEG): container finished" podID="cbd59738-5074-4422-913f-1d9e55ead178" containerID="cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e" exitCode=0 Nov 03 03:41:53.400735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.400599 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" Nov 03 03:41:53.400735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.400600 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" event={"ID":"cbd59738-5074-4422-913f-1d9e55ead178","Type":"ContainerDied","Data":"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e"} Nov 03 03:41:53.400735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.400712 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6" event={"ID":"cbd59738-5074-4422-913f-1d9e55ead178","Type":"ContainerDied","Data":"bdab0d926dd192197270d6d9b751d05a59ae3ef02fd7b2a048000826fc1c0552"} Nov 03 03:41:53.400735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.400732 5275 scope.go:122] "RemoveContainer" containerID="cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e" Nov 03 03:41:53.411919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.411894 5275 scope.go:122] "RemoveContainer" containerID="cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e" Nov 03 03:41:53.412230 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:41:53.412210 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e\": container with ID starting with cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e not found: ID does not exist" containerID="cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e" Nov 03 03:41:53.412305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.412239 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e"} err="failed to get container status \"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e\": rpc error: code = NotFound desc = could not find container \"cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e\": container with ID starting with cc9e87ad2ff1c2cee20f6fbf8e53bb60fe727a7431bb68a0ffaf33e3a663948e not found: ID does not exist" Nov 03 03:41:53.566201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.566085 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:41:53.569899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:53.569873 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2rwq6"] Nov 03 03:41:54.323368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:54.323343 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-znlwv_59ec2be7-fff9-4367-b67e-83e32e0f0586/payload-processing/0.log" Nov 03 03:41:54.336877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:54.336833 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:41:54.579365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:54.579278 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-znlwv_59ec2be7-fff9-4367-b67e-83e32e0f0586/payload-processing/0.log" Nov 03 03:41:54.813342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:54.813305 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj_a09fb47d-651c-4555-b026-1425f65b4148/payload-processing/0.log" Nov 03 03:41:54.985536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:54.985444 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd59738-5074-4422-913f-1d9e55ead178" path="/var/lib/kubelet/pods/cbd59738-5074-4422-913f-1d9e55ead178/volumes" Nov 03 03:41:55.343085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:55.343050 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:41:59.285773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.285734 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:41:59.286753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.286727 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cbd59738-5074-4422-913f-1d9e55ead178" containerName="discovery" Nov 03 03:41:59.286753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.286754 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd59738-5074-4422-913f-1d9e55ead178" containerName="discovery" Nov 03 03:41:59.286980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.286961 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd59738-5074-4422-913f-1d9e55ead178" containerName="discovery" Nov 03 03:41:59.291832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.291811 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.347704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.347672 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a6b9ff\"/\"openshift-service-ca.crt\"" Nov 03 03:41:59.348098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.348077 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a6b9ff\"/\"kube-root-ca.crt\"" Nov 03 03:41:59.368040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368006 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.368248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368050 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.368248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368147 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.368248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368215 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlsh\" (UniqueName: \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.368370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368303 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.368370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.368337 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.396923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.396841 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a6b9ff\"/\"ipp-route-a6b9ff-kserve-self-signed-certs\"" Nov 03 03:41:59.396923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.396920 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a6b9ff\"/\"default-dockercfg-c82rt\"" Nov 03 03:41:59.399982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.399957 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:41:59.470044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470011 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470076 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470188 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470214 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470251 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470321 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlsh\" (UniqueName: \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470507 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470539 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.470594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.470566 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.472731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.472702 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.472905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.472883 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.531366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.531325 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlsh\" (UniqueName: \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\") pod \"ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.604208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.604120 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:41:59.985177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:41:59.985146 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:41:59.986153 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:41:59.986127 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcad05144_5d31_4641_b471_395f7377e6fc.slice/crio-130aad5ef611f8a44c4102b87c427b766bfdb263f2d26c3aa7b7a9752da97b54 WatchSource:0}: Error finding container 130aad5ef611f8a44c4102b87c427b766bfdb263f2d26c3aa7b7a9752da97b54: Status 404 returned error can't find the container with id 130aad5ef611f8a44c4102b87c427b766bfdb263f2d26c3aa7b7a9752da97b54 Nov 03 03:42:00.466740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:00.466703 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" event={"ID":"cad05144-5d31-4641-b471-395f7377e6fc","Type":"ContainerStarted","Data":"fff49520e48c3cdcf7e22d40aa6109471d8f56472fe397ce13995651a5d85175"} Nov 03 03:42:00.466740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:00.466751 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" event={"ID":"cad05144-5d31-4641-b471-395f7377e6fc","Type":"ContainerStarted","Data":"130aad5ef611f8a44c4102b87c427b766bfdb263f2d26c3aa7b7a9752da97b54"} Nov 03 03:42:00.584897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:00.584818 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" podStartSLOduration=1.5848044890000001 podStartE2EDuration="1.584804489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:00.583196594 +0000 UTC m=+2411.735885017" watchObservedRunningTime="2026-11-03 03:42:00.584804489 +0000 UTC m=+2411.737492911" Nov 03 03:42:00.605161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:00.605129 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:42:02.330440 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:42:02.330406 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:42:09.605129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:09.605089 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:42:09.618277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:09.618242 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:42:10.543721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:10.543685 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:42:24.621440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.621395 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:42:24.625837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.625808 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.681906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.678387 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3480a2-openshift-default-dockercfg-xml9w\"" Nov 03 03:42:24.701813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.701775 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:42:24.725170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725130 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725187 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725335 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725472 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725531 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbzh\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725666 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725739 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.725820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.725798 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831591 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831668 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831709 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbzh\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831911 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.831975 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.832004 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.832074 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.832643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.832104 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.837322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.836633 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.837322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.836988 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.840243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.840206 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.840560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.840533 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.841161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.841133 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.842038 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.841990 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.845587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.845563 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.902546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.902439 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbzh\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.902902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.902833 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\") pod \"e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:24.947325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:24.947280 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:25.389677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.389650 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:42:25.392811 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:42:25.392781 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48f0011_819d_4ce3_be7e_de2f91107715.slice/crio-4eccb3ca94d250017ced2695eefdd223f4721e6a02c9c91887577ae0f655119f WatchSource:0}: Error finding container 4eccb3ca94d250017ced2695eefdd223f4721e6a02c9c91887577ae0f655119f: Status 404 returned error can't find the container with id 4eccb3ca94d250017ced2695eefdd223f4721e6a02c9c91887577ae0f655119f Nov 03 03:42:25.395586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.395544 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:25.395705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.395616 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:25.395705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.395649 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:25.654092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.653999 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerStarted","Data":"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e"} Nov 03 03:42:25.654493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.654118 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerStarted","Data":"4eccb3ca94d250017ced2695eefdd223f4721e6a02c9c91887577ae0f655119f"} Nov 03 03:42:25.775894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.775770 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podStartSLOduration=1.775751485 podStartE2EDuration="1.775751485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:25.775579591 +0000 UTC m=+2436.928268025" watchObservedRunningTime="2026-11-03 03:42:25.775751485 +0000 UTC m=+2436.928439907" Nov 03 03:42:25.948341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:25.948261 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:26.953892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:26.953829 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:27.673504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.673466 5275 generic.go:356] "Generic (PLEG): container finished" podID="192a40d2-037f-4757-919b-dc11688e9b19" containerID="834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d" exitCode=0 Nov 03 03:42:27.673752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.673548 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerDied","Data":"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d"} Nov 03 03:42:27.673913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.673896 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:27.674084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.674060 5275 scope.go:122] "RemoveContainer" containerID="834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d" Nov 03 03:42:27.675153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.674838 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:42:27.675765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.675731 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:27.675895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.675809 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:27.675895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:27.675863 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:42:28.683742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:28.683712 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerStarted","Data":"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753"} Nov 03 03:42:29.202775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:29.202740 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:42:29.203027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:29.203005 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 03:42:29.203089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:29.203052 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 03:42:30.202546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:30.202509 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 03:42:30.203000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:30.202566 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 03:42:31.202326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:31.202284 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 03:42:31.202541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:31.202360 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 03:42:32.203066 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.203028 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 03:42:32.203511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.203085 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 03:42:32.345869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.345070 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:42:32.354656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.354622 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.429892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.407974 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3480a2-dockercfg-gv2g8\"" Nov 03 03:42:32.429892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.408417 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:42:32.505146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.473677 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3480a2\"" Nov 03 03:42:32.505146 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:42:32.504154 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:42:32.517413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.517383 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:42:32.522062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.522037 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.571877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.550166 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwctj\" (UniqueName: \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.571877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.550284 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.580505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.580467 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:42:32.656873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.656725 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwctj\" (UniqueName: \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.656873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.656813 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x49t\" (UniqueName: \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.657159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.656900 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.657159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.656939 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.661881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.657895 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.718832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.718789 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwctj\" (UniqueName: \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\") pod \"payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.766494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.766146 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:32.767877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.767694 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x49t\" (UniqueName: \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.767877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.767789 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.772532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.769183 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.831519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.831475 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x49t\" (UniqueName: \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\") pod \"payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:32.839713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:32.839683 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:33.179300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.179267 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:42:33.180300 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:42:33.180269 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a33a003_b652_4b11_bf56_f8c8b254d3b5.slice/crio-68f4dcbfb759041b08463169b6cd6839f575cb617b0f1fff57551d612a318c58 WatchSource:0}: Error finding container 68f4dcbfb759041b08463169b6cd6839f575cb617b0f1fff57551d612a318c58: Status 404 returned error can't find the container with id 68f4dcbfb759041b08463169b6cd6839f575cb617b0f1fff57551d612a318c58 Nov 03 03:42:33.202992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.202958 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 03:42:33.203121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.203037 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 03:42:33.303629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.303560 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:42:33.306208 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:42:33.306159 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89d22e0_bb48_403c_abdc_9fc2bbce046b.slice/crio-75de59b08a320d2bfb1985334bad81719ad5e97dfdc388cb8bb1b9a2a10da7e9 WatchSource:0}: Error finding container 75de59b08a320d2bfb1985334bad81719ad5e97dfdc388cb8bb1b9a2a10da7e9: Status 404 returned error can't find the container with id 75de59b08a320d2bfb1985334bad81719ad5e97dfdc388cb8bb1b9a2a10da7e9 Nov 03 03:42:33.725129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.725090 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" event={"ID":"f89d22e0-bb48-403c-abdc-9fc2bbce046b","Type":"ContainerStarted","Data":"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4"} Nov 03 03:42:33.725387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.725369 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:42:33.725550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.725524 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" event={"ID":"f89d22e0-bb48-403c-abdc-9fc2bbce046b","Type":"ContainerStarted","Data":"75de59b08a320d2bfb1985334bad81719ad5e97dfdc388cb8bb1b9a2a10da7e9"} Nov 03 03:42:33.727033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.727006 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" event={"ID":"2a33a003-b652-4b11-bf56-f8c8b254d3b5","Type":"ContainerStarted","Data":"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e"} Nov 03 03:42:33.727148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.727061 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" event={"ID":"2a33a003-b652-4b11-bf56-f8c8b254d3b5","Type":"ContainerStarted","Data":"68f4dcbfb759041b08463169b6cd6839f575cb617b0f1fff57551d612a318c58"} Nov 03 03:42:33.727148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.727090 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:33.843643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.843577 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" podStartSLOduration=1.8435580520000001 podStartE2EDuration="1.843558052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:33.8402898 +0000 UTC m=+2444.992978223" watchObservedRunningTime="2026-11-03 03:42:33.843558052 +0000 UTC m=+2444.996246474" Nov 03 03:42:33.959070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:33.958978 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" podStartSLOduration=1.958954401 podStartE2EDuration="1.958954401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:33.957192883 +0000 UTC m=+2445.109881304" watchObservedRunningTime="2026-11-03 03:42:33.958954401 +0000 UTC m=+2445.111642823" Nov 03 03:42:34.207589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:34.207551 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:42:34.208097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:34.207874 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:42:34.208598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:34.208578 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:42:44.737829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:44.737754 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:42:44.737829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:42:44.737820 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:43:02.330601 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:43:02.330559 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:43:11.413352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:11.413315 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:43:11.413827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:11.413531 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" containerID="cri-o://90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b" gracePeriod=30 Nov 03 03:43:16.689204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.689135 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:43:16.732994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.732955 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733134 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733202 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733247 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733270 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:16.733401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733297 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733398 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733439 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733474 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733533 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\") pod \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\" (UID: \"556ab9f8-6f92-4038-b0ef-15d4dc8228f3\") " Nov 03 03:43:16.733831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.733804 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:16.734085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.734062 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:16.734182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.734081 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:16.734445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.734420 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:16.736209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736183 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.736312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736218 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.736312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736233 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.736312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736247 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.736312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736261 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.736312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.736188 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:16.737135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.737105 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "kube-api-access-bfq8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:16.737592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.737563 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:43:16.738481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.738453 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token" pod "556ab9f8-6f92-4038-b0ef-15d4dc8228f3" (UID: "556ab9f8-6f92-4038-b0ef-15d4dc8228f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:16.837499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.837464 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfq8d\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-kube-api-access-bfq8d\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.837499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.837498 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.837758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.837509 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:16.837758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:16.837517 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556ab9f8-6f92-4038-b0ef-15d4dc8228f3-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:17.035451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.035415 5275 generic.go:356] "Generic (PLEG): container finished" podID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerID="90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b" exitCode=0 Nov 03 03:43:17.035629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.035477 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" Nov 03 03:43:17.035629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.035501 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerDied","Data":"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b"} Nov 03 03:43:17.035629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.035560 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2" event={"ID":"556ab9f8-6f92-4038-b0ef-15d4dc8228f3","Type":"ContainerDied","Data":"09a32eb2256d1dacbaf554326666bdc10e1b8acbb81343a35fdc4a5ece7662bb"} Nov 03 03:43:17.035629 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.035582 5275 scope.go:122] "RemoveContainer" containerID="90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b" Nov 03 03:43:17.046020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.045999 5275 scope.go:122] "RemoveContainer" containerID="bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f" Nov 03 03:43:17.056129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.056104 5275 scope.go:122] "RemoveContainer" containerID="90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b" Nov 03 03:43:17.056451 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:43:17.056429 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b\": container with ID starting with 90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b not found: ID does not exist" containerID="90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b" Nov 03 03:43:17.056532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.056466 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b"} err="failed to get container status \"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b\": rpc error: code = NotFound desc = could not find container \"90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b\": container with ID starting with 90fa248422b52e5e3e6352d61b957c6abb64f92d97f48caf637b07f4e0b5c72b not found: ID does not exist" Nov 03 03:43:17.056532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.056494 5275 scope.go:122] "RemoveContainer" containerID="bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f" Nov 03 03:43:17.056765 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:43:17.056738 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f\": container with ID starting with bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f not found: ID does not exist" containerID="bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f" Nov 03 03:43:17.056815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.056773 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f"} err="failed to get container status \"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f\": rpc error: code = NotFound desc = could not find container \"bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f\": container with ID starting with bef8aaeab3feecb40f0db6bb736043bc276822b263ddbe7f022fe696a056254f not found: ID does not exist" Nov 03 03:43:17.216915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.216879 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:43:17.219717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:17.219687 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6e985-openshift-default-65f44d5697-7l4c2"] Nov 03 03:43:18.985687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:18.985652 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" path="/var/lib/kubelet/pods/556ab9f8-6f92-4038-b0ef-15d4dc8228f3/volumes" Nov 03 03:43:19.673218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673176 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:43:19.673726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673709 5275 cpu_manager.go:414] "containerMap: removing container" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.673726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673725 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.673929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673761 5275 cpu_manager.go:414] "containerMap: removing container" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.673929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673768 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.673929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673871 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.673929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.673883 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="556ab9f8-6f92-4038-b0ef-15d4dc8228f3" containerName="istio-proxy" Nov 03 03:43:19.678908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.678884 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.744881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.744241 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a9e5cd-openshift-default-dockercfg-zjs8b\"" Nov 03 03:43:19.770573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.770532 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.770913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.770893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.771184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.771116 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4549z\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.771387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.771372 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.783062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.777406 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.784556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.784475 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.784689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.784623 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.784758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.784686 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.784758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.784737 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.800355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.800316 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:43:19.886324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886286 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4549z\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886343 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886484 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886566 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886654 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886727 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886778 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886809 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886821 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.886961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.886920 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.887054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.887028 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.887108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.887055 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.887162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.887139 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.887243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.887216 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.889229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.889206 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.889324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.889305 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.947496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.947415 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.947636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.947612 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4549z\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\") pod \"e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:19.993978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:19.993942 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:20.437913 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:43:20.437883 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f9c7aa_fff2_4a4e_acff_0da6c71209c0.slice/crio-e6d00ace46499c7de123c7342997da94d238427714de71128e9b8bdf6b497d55 WatchSource:0}: Error finding container e6d00ace46499c7de123c7342997da94d238427714de71128e9b8bdf6b497d55: Status 404 returned error can't find the container with id e6d00ace46499c7de123c7342997da94d238427714de71128e9b8bdf6b497d55 Nov 03 03:43:20.438357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:20.438334 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:43:20.440623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:20.440586 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:20.440719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:20.440667 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:20.440719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:20.440710 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:21.078062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:21.078023 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerStarted","Data":"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a"} Nov 03 03:43:21.078575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:21.078080 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerStarted","Data":"e6d00ace46499c7de123c7342997da94d238427714de71128e9b8bdf6b497d55"} Nov 03 03:43:21.205718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:21.205665 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podStartSLOduration=2.205652163 podStartE2EDuration="2.205652163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:21.202483799 +0000 UTC m=+2492.355172246" watchObservedRunningTime="2026-11-03 03:43:21.205652163 +0000 UTC m=+2492.358340585" Nov 03 03:43:21.994422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:21.994384 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:22.000321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:22.000298 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:22.084915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:22.084883 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:22.086072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:22.086049 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:43:32.330650 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:43:32.330614 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:43:42.956650 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:42.956612 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:43:42.957208 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:42.956880 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" containerID="cri-o://fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2" gracePeriod=30 Nov 03 03:43:46.620891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:46.620836 5275 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:43:46.621382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:46.620933 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:43:48.224689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.224652 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:43:48.274837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.274808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.274884 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.274927 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.274983 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275046 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275117 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275131 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:48.275291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275151 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275186 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275216 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\") pod \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\" (UID: \"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a\") " Nov 03 03:43:48.275540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275287 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:48.275540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275465 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:48.275733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275710 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:48.275733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275714 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:48.276005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275935 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.276005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275954 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.276005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275965 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.276005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275974 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.276005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.275982 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.277345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277315 5275 generic.go:356] "Generic (PLEG): container finished" podID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerID="fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2" exitCode=0 Nov 03 03:43:48.277345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277358 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" event={"ID":"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a","Type":"ContainerDied","Data":"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2"} Nov 03 03:43:48.277559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277413 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" event={"ID":"ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a","Type":"ContainerDied","Data":"166e4e10b8c3e6f8b33f5056ad1652147efc76034e5984478cd90ee1321c5ea6"} Nov 03 03:43:48.277559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277446 5275 scope.go:122] "RemoveContainer" containerID="fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2" Nov 03 03:43:48.277681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277597 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj" Nov 03 03:43:48.277681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277635 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:43:48.277931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:43:48.278046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.277928 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:48.279244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.279221 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf" pod "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" (UID: "ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a"). InnerVolumeSpecName "kube-api-access-nrkbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:48.306817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.306793 5275 scope.go:122] "RemoveContainer" containerID="fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2" Nov 03 03:43:48.307221 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:43:48.307202 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2\": container with ID starting with fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2 not found: ID does not exist" containerID="fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2" Nov 03 03:43:48.307281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.307234 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2"} err="failed to get container status \"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2\": rpc error: code = NotFound desc = could not find container \"fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2\": container with ID starting with fb9f73e36cf577ff8c5172e1cd3244ddfafc46df1d4cda5ef08ca7a9af1cf8f2 not found: ID does not exist" Nov 03 03:43:48.376941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.376909 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.376941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.376938 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-kube-api-access-nrkbf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.376941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.376947 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.377286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.376957 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:48.702258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.702219 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:43:48.706216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.706183 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-vwszj"] Nov 03 03:43:48.985943 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:48.985866 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" path="/var/lib/kubelet/pods/ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a/volumes" Nov 03 03:43:52.787653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.787558 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:43:52.788433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.788409 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" Nov 03 03:43:52.788543 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.788435 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" Nov 03 03:43:52.788595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.788584 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfc344b-d698-49d7-b9c1-d0a0bf4aa49a" containerName="istio-proxy" Nov 03 03:43:52.793913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.793891 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.848876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.846943 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2be7ec-openshift-default-dockercfg-d9gnn\"" Nov 03 03:43:52.918885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.913988 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:43:52.972993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.972966 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.973427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.973371 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbg8w\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.973591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.973580 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.973734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.973723 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.973938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.973921 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.974279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.974265 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.974532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.974519 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.974699 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.974688 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:52.976469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:52.974836 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078586 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078640 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbg8w\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078711 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078742 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.078961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078939 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.078968 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079008 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079067 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079570 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079631 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079686 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.079944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079839 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.080010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.079977 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.081499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.081470 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.081678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.081659 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.150925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.150894 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.154344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.154306 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbg8w\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\") pod \"e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.408442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.408345 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:53.845152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.845118 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:43:53.846385 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:43:53.846360 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6babc80_bdb9_459b_be16_991b3f4a2380.slice/crio-eafd5cb965485bce8f1c5417d427cc338eb2b74baae3e4e67f06d5843a99901f WatchSource:0}: Error finding container eafd5cb965485bce8f1c5417d427cc338eb2b74baae3e4e67f06d5843a99901f: Status 404 returned error can't find the container with id eafd5cb965485bce8f1c5417d427cc338eb2b74baae3e4e67f06d5843a99901f Nov 03 03:43:53.849136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.849101 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:53.849226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.849173 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:53.849226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:53.849199 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:43:54.324068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:54.324036 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerStarted","Data":"d11348047229602a94be7df2844e3bd4b3a0abb96a4cc18b2ba814758edf652f"} Nov 03 03:43:54.324257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:54.324092 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerStarted","Data":"eafd5cb965485bce8f1c5417d427cc338eb2b74baae3e4e67f06d5843a99901f"} Nov 03 03:43:54.409447 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:54.409412 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:54.416333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:54.416303 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:54.453122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:54.453051 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podStartSLOduration=2.453032506 podStartE2EDuration="2.453032506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:54.450771644 +0000 UTC m=+2525.603460065" watchObservedRunningTime="2026-11-03 03:43:54.453032506 +0000 UTC m=+2525.605720929" Nov 03 03:43:55.330672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:55.330639 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:43:55.331544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:43:55.331526 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:44:02.331292 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:44:02.331256 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:44:11.516862 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.516802 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:44:11.521019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.520996 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.585670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.585630 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:44:11.601945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.600392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2l9\" (UniqueName: \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.601945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.600576 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.626877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.624754 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9b4cb2\"" Nov 03 03:44:11.626877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.624923 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9b4cb2-dockercfg-2s2tk\"" Nov 03 03:44:11.709879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.709659 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.709879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.709801 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2l9\" (UniqueName: \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.717882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.711118 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.757869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.755909 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:44:11.804081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.804047 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:44:11.804421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.804366 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:11.805568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.805536 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2l9\" (UniqueName: \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\") pod \"payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.839377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.839049 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:11.914371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.914328 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:11.914564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:11.914541 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7qq\" (UniqueName: \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.015837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.015796 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.016049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.015989 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7qq\" (UniqueName: \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.017188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.017157 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.098921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.096438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7qq\" (UniqueName: \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\") pod \"payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.126446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.126409 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:12.287987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.287590 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:44:12.504088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.504045 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" event={"ID":"e286c718-9092-40bd-9f17-7d2a67c52176","Type":"ContainerStarted","Data":"8461dcd33df0ccdd831d1f781ea1c6e1944d0a2663942f4875938e647642e0a2"} Nov 03 03:44:12.504296 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.504269 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:12.504369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.504303 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" event={"ID":"e286c718-9092-40bd-9f17-7d2a67c52176","Type":"ContainerStarted","Data":"6fe5ed99bd04ed067d124e53d15352d3722f90505cc8a18c83fab0110b9a6f9a"} Nov 03 03:44:12.600175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.599317 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:44:12.654444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:12.654388 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" podStartSLOduration=1.6543726749999998 podStartE2EDuration="1.654372675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:12.649561655 +0000 UTC m=+2543.802250077" watchObservedRunningTime="2026-11-03 03:44:12.654372675 +0000 UTC m=+2543.807061095" Nov 03 03:44:13.515256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:13.515215 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" event={"ID":"40809b23-00a6-4ca7-b7d5-57ecad115ef6","Type":"ContainerStarted","Data":"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc"} Nov 03 03:44:13.515491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:13.515274 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" event={"ID":"40809b23-00a6-4ca7-b7d5-57ecad115ef6","Type":"ContainerStarted","Data":"8479d93e234cf616c6cd1a4543a7dd312c947d2692b8c6aab2f7da8fcfcf9e3f"} Nov 03 03:44:13.633891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:13.631441 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" podStartSLOduration=2.631422619 podStartE2EDuration="2.631422619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:13.630492638 +0000 UTC m=+2544.783181058" watchObservedRunningTime="2026-11-03 03:44:13.631422619 +0000 UTC m=+2544.784111042" Nov 03 03:44:14.522983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:14.522954 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:16.539667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.539635 5275 generic.go:356] "Generic (PLEG): container finished" podID="f48f0011-819d-4ce3-be7e-de2f91107715" containerID="914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e" exitCode=0 Nov 03 03:44:16.540249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.539714 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerDied","Data":"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e"} Nov 03 03:44:16.540322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.540273 5275 scope.go:122] "RemoveContainer" containerID="914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e" Nov 03 03:44:16.542210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.542177 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:16.542316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.542299 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:16.542363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:16.542348 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:17.557873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:17.553249 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerStarted","Data":"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a"} Nov 03 03:44:17.947514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:17.947420 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:44:17.947727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:17.947703 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 03:44:17.947809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:17.947752 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 03:44:18.948024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:18.947988 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 03:44:18.948428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:18.948043 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 03:44:19.948291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:19.948258 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 03:44:19.948768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:19.948336 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 03:44:20.947869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:20.947784 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 03:44:20.948097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:20.947901 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 03:44:21.948054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:21.948011 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 03:44:21.948573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:21.948074 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 03:44:22.953201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:22.953169 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:44:22.953605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:22.953583 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:44:22.954302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:22.954282 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:44:23.518404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:23.518374 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:44:25.532730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:25.532697 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:44:32.330894 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:44:32.330829 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:44:42.181061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:42.181033 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-znlwv_59ec2be7-fff9-4367-b67e-83e32e0f0586/payload-processing/0.log" Nov 03 03:44:42.431908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:42.431807 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-znlwv_59ec2be7-fff9-4367-b67e-83e32e0f0586/payload-processing/0.log" Nov 03 03:44:42.667594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:42.667558 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj_a09fb47d-651c-4555-b026-1425f65b4148/payload-processing/0.log" Nov 03 03:44:42.943122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:42.943095 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj_a09fb47d-651c-4555-b026-1425f65b4148/payload-processing/0.log" Nov 03 03:44:48.818174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:48.818127 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:44:58.879099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.879059 5275 generic.go:356] "Generic (PLEG): container finished" podID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerID="18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a" exitCode=0 Nov 03 03:44:58.879653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.879129 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerDied","Data":"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a"} Nov 03 03:44:58.879653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.879593 5275 scope.go:122] "RemoveContainer" containerID="18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a" Nov 03 03:44:58.881251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.881221 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:58.881354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.881293 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:58.881354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:58.881327 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:44:59.888823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:59.888787 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerStarted","Data":"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9"} Nov 03 03:44:59.994780 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:59.994740 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:44:59.995013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:59.994983 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:44:59.995075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:44:59.995042 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:00.196619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.196522 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:00.200659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.200622 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:00.201239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.201210 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.213470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.213444 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:00.217741 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.217709 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.220709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.220671 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.221095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.221075 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:00.226301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.226279 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:00.226452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.226433 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.229172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.229153 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:00.229301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.229280 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.231725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.231706 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:00.231867 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.231832 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.234473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.234455 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.234675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.234655 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:00.237519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.237502 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.239300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.239283 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:00.243822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.243782 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:00.271867 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.271172 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:45:00.272063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.271951 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:45:00.315098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.314948 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:00.315098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.315020 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:00.315249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.315222 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:00.317377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.317352 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.317601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.317582 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.317726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.317713 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vn2\" (UniqueName: \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.317871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.317839 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdfg\" (UniqueName: \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.318014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klx8g\" (UniqueName: \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.318182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318168 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.318282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318270 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.318371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318359 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvcv\" (UniqueName: \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.318475 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318462 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnz2\" (UniqueName: \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.318590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318575 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m28t\" (UniqueName: \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.318691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318678 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.318831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.318819 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.321108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.319302 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkl7k\" (UniqueName: \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.321308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.321279 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8ll\" (UniqueName: \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.321587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.321566 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.321817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.321795 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.345895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.340986 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:00.345895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.341989 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:45:00.417868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.417746 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:00.417868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.417809 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424536 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnz2\" (UniqueName: \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424595 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m28t\" (UniqueName: \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424627 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424706 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424757 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkl7k\" (UniqueName: \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424783 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8ll\" (UniqueName: \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424822 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424880 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424935 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.424983 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425025 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vn2\" (UniqueName: \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425076 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdfg\" (UniqueName: \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425113 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klx8g\" (UniqueName: \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425203 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425236 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.425504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.425261 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvcv\" (UniqueName: \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.427158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.427039 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.429263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.428367 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.429263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.428462 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.429263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.428586 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.429263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.429003 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.429263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.429203 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.435242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.430136 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.435242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.430549 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.499395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.495007 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnz2\" (UniqueName: \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\") pod \"maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.499395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.497880 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkl7k\" (UniqueName: \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\") pod \"maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.499395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.498575 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8ll\" (UniqueName: \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\") pod \"maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.499395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.499326 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vn2\" (UniqueName: \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\") pod \"maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.500654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.500327 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klx8g\" (UniqueName: \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\") pod \"maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.500654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.500612 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m28t\" (UniqueName: \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.501435 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.501129 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdfg\" (UniqueName: \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.502368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.502333 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvcv\" (UniqueName: \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\") pod \"maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.520830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.520799 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:00.533101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.533058 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:00.546593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.546561 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:00.588302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.575227 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:00.610887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.609684 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:00.630395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.630355 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:00.644138 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.644095 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:00.713889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.705531 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:00.995920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.994391 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:45:00.995920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:00.994448 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:01.036089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.035288 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:01.036245 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:45:01.036153 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6db340_11fa_4f82_bfbe_901af3acb2d3.slice/crio-5337e52490af1b5fb58ef63555492f7a481059e4190dbc9f79cec5a496c22e84 WatchSource:0}: Error finding container 5337e52490af1b5fb58ef63555492f7a481059e4190dbc9f79cec5a496c22e84: Status 404 returned error can't find the container with id 5337e52490af1b5fb58ef63555492f7a481059e4190dbc9f79cec5a496c22e84 Nov 03 03:45:01.246114 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:45:01.246079 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48fa82b_0be1_4b1b_9bf5_326b6ebd6464.slice/crio-d2cf7dedd18294dbb9c87ac5ea1a3991fcda2c9561b9d3808f8841f0abadb566 WatchSource:0}: Error finding container d2cf7dedd18294dbb9c87ac5ea1a3991fcda2c9561b9d3808f8841f0abadb566: Status 404 returned error can't find the container with id d2cf7dedd18294dbb9c87ac5ea1a3991fcda2c9561b9d3808f8841f0abadb566 Nov 03 03:45:01.247185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.247160 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:01.289061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.289039 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:01.301089 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:45:01.301035 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59137ac2_e3c6_490a_9566_114a9f2b1e8d.slice/crio-8da3df6da79fab63899922ece5f3abbfdc5e8b20980bc0700182804ffe64aade WatchSource:0}: Error finding container 8da3df6da79fab63899922ece5f3abbfdc5e8b20980bc0700182804ffe64aade: Status 404 returned error can't find the container with id 8da3df6da79fab63899922ece5f3abbfdc5e8b20980bc0700182804ffe64aade Nov 03 03:45:01.348453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.346727 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:01.416926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.415020 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:01.434010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.433983 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:01.439934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.439910 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:01.490760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.489591 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:01.940083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.940046 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerStarted","Data":"f616b78cd80eccd66e88a92f18a5ad6909fe26ea07f5ef9f1081087f9454bda3"} Nov 03 03:45:01.940388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.940109 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerStarted","Data":"8da3df6da79fab63899922ece5f3abbfdc5e8b20980bc0700182804ffe64aade"} Nov 03 03:45:01.941672 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.941627 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerStarted","Data":"3b5ade924d144f151c6f45ad0e85774ac11079b15a03af83bd0bdda92518ab88"} Nov 03 03:45:01.943034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.943010 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerStarted","Data":"84a85fe7a2a4ccc3a7643ff57fa8547813564090bfdee7e6f202c164e17e3efc"} Nov 03 03:45:01.945172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.944525 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerStarted","Data":"f9b6316bc8e1ae6347218e80cf32340ab072de3df7be7786fb17eadec37c8c6e"} Nov 03 03:45:01.945172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.944558 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerStarted","Data":"5337e52490af1b5fb58ef63555492f7a481059e4190dbc9f79cec5a496c22e84"} Nov 03 03:45:01.946021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.945997 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerStarted","Data":"cc15e3a8c622d4bcd6347f03ff9915deaa55314064263a3f3b8004d4eba41bb9"} Nov 03 03:45:01.946110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.946035 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerStarted","Data":"d2cf7dedd18294dbb9c87ac5ea1a3991fcda2c9561b9d3808f8841f0abadb566"} Nov 03 03:45:01.947159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.947138 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerStarted","Data":"b47cd51c326c76db65b024f36cb19e687b5c73024478770e6eb7bac8abab0722"} Nov 03 03:45:01.948267 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.948247 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerStarted","Data":"1f68e8b2c1c9f147685959cba9516a1de3a6b7dd54457f7ec1e22abd029848a3"} Nov 03 03:45:01.972211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.972167 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerStarted","Data":"580b7708122a2b22c35c57c428b3c6ad480b9da5d72b6719f7ecb750e0ad9808"} Nov 03 03:45:01.998414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.998380 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:45:01.998960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:01.998932 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:02.091976 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.091419 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" podStartSLOduration=2.091392318 podStartE2EDuration="2.091392318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:02.076940638 +0000 UTC m=+2593.229629063" watchObservedRunningTime="2026-11-03 03:45:02.091392318 +0000 UTC m=+2593.244080741" Nov 03 03:45:02.193997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.193310 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" podStartSLOduration=2.193294079 podStartE2EDuration="2.193294079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:02.192704598 +0000 UTC m=+2593.345393020" watchObservedRunningTime="2026-11-03 03:45:02.193294079 +0000 UTC m=+2593.345982501" Nov 03 03:45:02.313974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.313917 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" podStartSLOduration=2.3138994139999998 podStartE2EDuration="2.313899414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:02.310298429 +0000 UTC m=+2593.462986851" watchObservedRunningTime="2026-11-03 03:45:02.313899414 +0000 UTC m=+2593.466587833" Nov 03 03:45:02.331952 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:02.331866 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:45:02.988648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.988598 5275 generic.go:356] "Generic (PLEG): container finished" podID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerID="e5bab2bd7534a5ec8bedaff89d53c8524b66cc4666e3c46471f9daaf74acbc08" exitCode=60 Nov 03 03:45:02.989185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.989157 5275 scope.go:122] "RemoveContainer" containerID="e5bab2bd7534a5ec8bedaff89d53c8524b66cc4666e3c46471f9daaf74acbc08" Nov 03 03:45:02.990107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.989977 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerDied","Data":"e5bab2bd7534a5ec8bedaff89d53c8524b66cc4666e3c46471f9daaf74acbc08"} Nov 03 03:45:02.990820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.990797 5275 generic.go:356] "Generic (PLEG): container finished" podID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerID="e438d0770283573bcacd5728783b0c3ad9e1be54a1942e41fb4b6db09abc541a" exitCode=60 Nov 03 03:45:02.990968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.990906 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerDied","Data":"e438d0770283573bcacd5728783b0c3ad9e1be54a1942e41fb4b6db09abc541a"} Nov 03 03:45:02.991150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.991131 5275 scope.go:122] "RemoveContainer" containerID="e438d0770283573bcacd5728783b0c3ad9e1be54a1942e41fb4b6db09abc541a" Nov 03 03:45:02.992917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.992894 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerID="f9b6316bc8e1ae6347218e80cf32340ab072de3df7be7786fb17eadec37c8c6e" exitCode=60 Nov 03 03:45:02.993029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.992995 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerDied","Data":"f9b6316bc8e1ae6347218e80cf32340ab072de3df7be7786fb17eadec37c8c6e"} Nov 03 03:45:02.993255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.993212 5275 scope.go:122] "RemoveContainer" containerID="f9b6316bc8e1ae6347218e80cf32340ab072de3df7be7786fb17eadec37c8c6e" Nov 03 03:45:02.994899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.994871 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:45:02.994970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.994919 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:02.995279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.995244 5275 generic.go:356] "Generic (PLEG): container finished" podID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerID="cc15e3a8c622d4bcd6347f03ff9915deaa55314064263a3f3b8004d4eba41bb9" exitCode=60 Nov 03 03:45:02.995386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.995317 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerDied","Data":"cc15e3a8c622d4bcd6347f03ff9915deaa55314064263a3f3b8004d4eba41bb9"} Nov 03 03:45:02.995494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.995478 5275 scope.go:122] "RemoveContainer" containerID="cc15e3a8c622d4bcd6347f03ff9915deaa55314064263a3f3b8004d4eba41bb9" Nov 03 03:45:02.997049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.997031 5275 generic.go:356] "Generic (PLEG): container finished" podID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerID="f6a1b26c496c3273568e4089ae98be6cb937932a09e4bec67cfc4a318c2b70c8" exitCode=60 Nov 03 03:45:02.997237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.997216 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerDied","Data":"f6a1b26c496c3273568e4089ae98be6cb937932a09e4bec67cfc4a318c2b70c8"} Nov 03 03:45:02.997407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.997312 5275 scope.go:122] "RemoveContainer" containerID="f6a1b26c496c3273568e4089ae98be6cb937932a09e4bec67cfc4a318c2b70c8" Nov 03 03:45:02.998900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.998880 5275 generic.go:356] "Generic (PLEG): container finished" podID="6e55b5f7-e13d-405a-9813-378da3636323" containerID="a977966b01b5620b9f041043ebdc3d459d52c02527891c61c4ddc1afaa9479a3" exitCode=60 Nov 03 03:45:02.999253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.998980 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerDied","Data":"a977966b01b5620b9f041043ebdc3d459d52c02527891c61c4ddc1afaa9479a3"} Nov 03 03:45:02.999450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:02.999429 5275 scope.go:122] "RemoveContainer" containerID="a977966b01b5620b9f041043ebdc3d459d52c02527891c61c4ddc1afaa9479a3" Nov 03 03:45:03.001901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.001306 5275 generic.go:356] "Generic (PLEG): container finished" podID="6407137b-8c3d-4319-a853-a026612b6922" containerID="ca1421664ff967d841dbe471a5b95dca295f6aec13d55972fe349c816660857b" exitCode=60 Nov 03 03:45:03.001901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.001563 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerDied","Data":"ca1421664ff967d841dbe471a5b95dca295f6aec13d55972fe349c816660857b"} Nov 03 03:45:03.001901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.001619 5275 scope.go:122] "RemoveContainer" containerID="ca1421664ff967d841dbe471a5b95dca295f6aec13d55972fe349c816660857b" Nov 03 03:45:03.003496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.003476 5275 generic.go:356] "Generic (PLEG): container finished" podID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerID="f616b78cd80eccd66e88a92f18a5ad6909fe26ea07f5ef9f1081087f9454bda3" exitCode=60 Nov 03 03:45:03.003598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.003506 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerDied","Data":"f616b78cd80eccd66e88a92f18a5ad6909fe26ea07f5ef9f1081087f9454bda3"} Nov 03 03:45:03.003732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.003719 5275 scope.go:122] "RemoveContainer" containerID="f616b78cd80eccd66e88a92f18a5ad6909fe26ea07f5ef9f1081087f9454bda3" Nov 03 03:45:03.840540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.840502 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" podUID="cad05144-5d31-4641-b471-395f7377e6fc" containerName="main" containerID="cri-o://fff49520e48c3cdcf7e22d40aa6109471d8f56472fe397ce13995651a5d85175" gracePeriod=45 Nov 03 03:45:03.995327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.995282 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:45:03.995498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:03.995349 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:04.024333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.019971 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerStarted","Data":"f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a"} Nov 03 03:45:04.027133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.027067 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerStarted","Data":"bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67"} Nov 03 03:45:04.030446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.030376 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerStarted","Data":"09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54"} Nov 03 03:45:04.033966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.032796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerStarted","Data":"75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089"} Nov 03 03:45:04.035149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.035092 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerStarted","Data":"0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d"} Nov 03 03:45:04.037224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.037201 5275 generic.go:356] "Generic (PLEG): container finished" podID="cad05144-5d31-4641-b471-395f7377e6fc" containerID="fff49520e48c3cdcf7e22d40aa6109471d8f56472fe397ce13995651a5d85175" exitCode=0 Nov 03 03:45:04.037335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.037280 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" event={"ID":"cad05144-5d31-4641-b471-395f7377e6fc","Type":"ContainerDied","Data":"fff49520e48c3cdcf7e22d40aa6109471d8f56472fe397ce13995651a5d85175"} Nov 03 03:45:04.041806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.039453 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerStarted","Data":"6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a"} Nov 03 03:45:04.043047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.042545 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerStarted","Data":"ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d"} Nov 03 03:45:04.046341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.045350 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerStarted","Data":"9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364"} Nov 03 03:45:04.140876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.139488 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" podStartSLOduration=4.139469884 podStartE2EDuration="4.139469884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:04.138915037 +0000 UTC m=+2595.291603461" watchObservedRunningTime="2026-11-03 03:45:04.139469884 +0000 UTC m=+2595.292158307" Nov 03 03:45:04.287875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.272156 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" podStartSLOduration=4.27213474 podStartE2EDuration="4.27213474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:04.270330051 +0000 UTC m=+2595.423018500" watchObservedRunningTime="2026-11-03 03:45:04.27213474 +0000 UTC m=+2595.424823163" Nov 03 03:45:04.447888 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.447838 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:45:04.511779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.511726 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" podStartSLOduration=4.511710644 podStartE2EDuration="4.511710644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:04.395345442 +0000 UTC m=+2595.548033864" watchObservedRunningTime="2026-11-03 03:45:04.511710644 +0000 UTC m=+2595.664399064" Nov 03 03:45:04.542386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542353 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542466 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542531 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542587 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542640 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\" (UniqueName: \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542692 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\") pod \"cad05144-5d31-4641-b471-395f7377e6fc\" (UID: \"cad05144-5d31-4641-b471-395f7377e6fc\") " Nov 03 03:45:04.542897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542793 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:04.542897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.542814 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:04.543051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.543031 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:04.543286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.543238 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.543286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.543257 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.543286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.543267 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.544990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.544964 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:04.545356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.545331 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:04.545460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.545370 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh" pod "cad05144-5d31-4641-b471-395f7377e6fc" (UID: "cad05144-5d31-4641-b471-395f7377e6fc"). InnerVolumeSpecName "kube-api-access-ftlsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:04.645018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.644978 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlsh\" (UniqueName: \"kubernetes.io/projected/cad05144-5d31-4641-b471-395f7377e6fc-kube-api-access-ftlsh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.645018 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.645018 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad05144-5d31-4641-b471-395f7377e6fc-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.645263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.645032 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad05144-5d31-4641-b471-395f7377e6fc-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:04.994465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.994376 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 03:45:04.994465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:04.994435 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 03:45:05.004083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.004037 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" podStartSLOduration=5.004020989 podStartE2EDuration="5.004020989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:04.881169028 +0000 UTC m=+2596.033857449" watchObservedRunningTime="2026-11-03 03:45:05.004020989 +0000 UTC m=+2596.156709411" Nov 03 03:45:05.004232 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.004214 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" podStartSLOduration=5.004209176 podStartE2EDuration="5.004209176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:04.999726911 +0000 UTC m=+2596.152415332" watchObservedRunningTime="2026-11-03 03:45:05.004209176 +0000 UTC m=+2596.156897597" Nov 03 03:45:05.060838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.060804 5275 generic.go:356] "Generic (PLEG): container finished" podID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerID="9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364" exitCode=60 Nov 03 03:45:05.061356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.060910 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerDied","Data":"9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364"} Nov 03 03:45:05.061356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.060984 5275 scope.go:122] "RemoveContainer" containerID="e5bab2bd7534a5ec8bedaff89d53c8524b66cc4666e3c46471f9daaf74acbc08" Nov 03 03:45:05.061356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.061234 5275 scope.go:122] "RemoveContainer" containerID="9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364" Nov 03 03:45:05.061608 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.061586 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96_odh-ai-gateway-infra(4b497401-c25f-448f-a8ba-28cc0b3902c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" Nov 03 03:45:05.063701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.063680 5275 generic.go:356] "Generic (PLEG): container finished" podID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerID="f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a" exitCode=60 Nov 03 03:45:05.064004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.063983 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerDied","Data":"f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a"} Nov 03 03:45:05.064147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.064133 5275 scope.go:122] "RemoveContainer" containerID="f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a" Nov 03 03:45:05.064459 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.064433 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs_odh-ai-gateway-infra(2890d82d-69bd-4d36-a543-d9991ea53d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" Nov 03 03:45:05.066500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.066304 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerID="bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67" exitCode=60 Nov 03 03:45:05.066500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.066414 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerDied","Data":"bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67"} Nov 03 03:45:05.066776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.066759 5275 scope.go:122] "RemoveContainer" containerID="bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67" Nov 03 03:45:05.067160 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.067137 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp_odh-ai-gateway-infra(5c6db340-11fa-4f82-bfbe-901af3acb2d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" Nov 03 03:45:05.069130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.069114 5275 generic.go:356] "Generic (PLEG): container finished" podID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerID="09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54" exitCode=60 Nov 03 03:45:05.069297 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.069181 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerDied","Data":"09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54"} Nov 03 03:45:05.069499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.069476 5275 scope.go:122] "RemoveContainer" containerID="09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54" Nov 03 03:45:05.069838 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.069808 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9_odh-ai-gateway-infra(f48fa82b-0be1-4b1b-9bf5-326b6ebd6464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" Nov 03 03:45:05.071889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.071860 5275 generic.go:356] "Generic (PLEG): container finished" podID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerID="75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089" exitCode=60 Nov 03 03:45:05.071987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.071890 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerDied","Data":"75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089"} Nov 03 03:45:05.072243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.072219 5275 scope.go:122] "RemoveContainer" containerID="75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089" Nov 03 03:45:05.072586 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.072534 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc_odh-ai-gateway-infra(0694e9f5-4cd9-4deb-b3d9-03231046d9d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" Nov 03 03:45:05.080694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.079684 5275 generic.go:356] "Generic (PLEG): container finished" podID="6e55b5f7-e13d-405a-9813-378da3636323" containerID="0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d" exitCode=60 Nov 03 03:45:05.080694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.079887 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerDied","Data":"0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d"} Nov 03 03:45:05.080896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.080878 5275 scope.go:122] "RemoveContainer" containerID="0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d" Nov 03 03:45:05.081255 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.081230 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp_odh-ai-gateway-infra(6e55b5f7-e13d-405a-9813-378da3636323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" podUID="6e55b5f7-e13d-405a-9813-378da3636323" Nov 03 03:45:05.084541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.084520 5275 scope.go:122] "RemoveContainer" containerID="e438d0770283573bcacd5728783b0c3ad9e1be54a1942e41fb4b6db09abc541a" Nov 03 03:45:05.085053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.085033 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" Nov 03 03:45:05.085216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.085038 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf" event={"ID":"cad05144-5d31-4641-b471-395f7377e6fc","Type":"ContainerDied","Data":"130aad5ef611f8a44c4102b87c427b766bfdb263f2d26c3aa7b7a9752da97b54"} Nov 03 03:45:05.088384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.088360 5275 generic.go:356] "Generic (PLEG): container finished" podID="6407137b-8c3d-4319-a853-a026612b6922" containerID="6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a" exitCode=60 Nov 03 03:45:05.088483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.088445 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerDied","Data":"6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a"} Nov 03 03:45:05.088999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.088979 5275 scope.go:122] "RemoveContainer" containerID="6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a" Nov 03 03:45:05.089288 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.089261 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5_odh-ai-gateway-infra(6407137b-8c3d-4319-a853-a026612b6922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" podUID="6407137b-8c3d-4319-a853-a026612b6922" Nov 03 03:45:05.090773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.090748 5275 generic.go:356] "Generic (PLEG): container finished" podID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerID="ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d" exitCode=60 Nov 03 03:45:05.090895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.090780 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerDied","Data":"ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d"} Nov 03 03:45:05.091126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.091107 5275 scope.go:122] "RemoveContainer" containerID="ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d" Nov 03 03:45:05.091421 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:05.091394 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9_odh-ai-gateway-infra(59137ac2-e3c6-490a-9566-114a9f2b1e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" Nov 03 03:45:05.100227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.100193 5275 scope.go:122] "RemoveContainer" containerID="f9b6316bc8e1ae6347218e80cf32340ab072de3df7be7786fb17eadec37c8c6e" Nov 03 03:45:05.114304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.114282 5275 scope.go:122] "RemoveContainer" containerID="cc15e3a8c622d4bcd6347f03ff9915deaa55314064263a3f3b8004d4eba41bb9" Nov 03 03:45:05.135326 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.135303 5275 scope.go:122] "RemoveContainer" containerID="f6a1b26c496c3273568e4089ae98be6cb937932a09e4bec67cfc4a318c2b70c8" Nov 03 03:45:05.161501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.158239 5275 scope.go:122] "RemoveContainer" containerID="a977966b01b5620b9f041043ebdc3d459d52c02527891c61c4ddc1afaa9479a3" Nov 03 03:45:05.195484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.195456 5275 scope.go:122] "RemoveContainer" containerID="fff49520e48c3cdcf7e22d40aa6109471d8f56472fe397ce13995651a5d85175" Nov 03 03:45:05.208366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.208346 5275 scope.go:122] "RemoveContainer" containerID="ca1421664ff967d841dbe471a5b95dca295f6aec13d55972fe349c816660857b" Nov 03 03:45:05.231134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.227046 5275 scope.go:122] "RemoveContainer" containerID="f616b78cd80eccd66e88a92f18a5ad6909fe26ea07f5ef9f1081087f9454bda3" Nov 03 03:45:05.464736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.464705 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:45:05.469466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:05.469441 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a6b9ff/ipp-route-a6b9ff-kserve-5857f5f96f-kx2cf"] Nov 03 03:45:06.001076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.001046 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:45:06.001364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.001338 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:45:06.001999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.001980 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:45:06.104050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.104015 5275 scope.go:122] "RemoveContainer" containerID="75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089" Nov 03 03:45:06.104497 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.104378 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc_odh-ai-gateway-infra(0694e9f5-4cd9-4deb-b3d9-03231046d9d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" Nov 03 03:45:06.105766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.105746 5275 scope.go:122] "RemoveContainer" containerID="0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d" Nov 03 03:45:06.106072 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.106050 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp_odh-ai-gateway-infra(6e55b5f7-e13d-405a-9813-378da3636323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" podUID="6e55b5f7-e13d-405a-9813-378da3636323" Nov 03 03:45:06.108402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.108384 5275 scope.go:122] "RemoveContainer" containerID="6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a" Nov 03 03:45:06.108690 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.108667 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5_odh-ai-gateway-infra(6407137b-8c3d-4319-a853-a026612b6922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" podUID="6407137b-8c3d-4319-a853-a026612b6922" Nov 03 03:45:06.109996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.109980 5275 scope.go:122] "RemoveContainer" containerID="ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d" Nov 03 03:45:06.110255 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.110229 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9_odh-ai-gateway-infra(59137ac2-e3c6-490a-9566-114a9f2b1e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" Nov 03 03:45:06.111472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.111454 5275 scope.go:122] "RemoveContainer" containerID="9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364" Nov 03 03:45:06.111714 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.111696 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96_odh-ai-gateway-infra(4b497401-c25f-448f-a8ba-28cc0b3902c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" Nov 03 03:45:06.112959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.112942 5275 scope.go:122] "RemoveContainer" containerID="f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a" Nov 03 03:45:06.113153 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.113134 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs_odh-ai-gateway-infra(2890d82d-69bd-4d36-a543-d9991ea53d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" Nov 03 03:45:06.114581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.114566 5275 scope.go:122] "RemoveContainer" containerID="bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67" Nov 03 03:45:06.114783 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.114762 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp_odh-ai-gateway-infra(5c6db340-11fa-4f82-bfbe-901af3acb2d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" Nov 03 03:45:06.116108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:06.116091 5275 scope.go:122] "RemoveContainer" containerID="09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54" Nov 03 03:45:06.116275 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:06.116254 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9_odh-ai-gateway-infra(f48fa82b-0be1-4b1b-9bf5-326b6ebd6464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" Nov 03 03:45:07.009264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:07.009223 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad05144-5d31-4641-b471-395f7377e6fc" path="/var/lib/kubelet/pods/cad05144-5d31-4641-b471-395f7377e6fc/volumes" Nov 03 03:45:14.978254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:14.978223 5275 scope.go:122] "RemoveContainer" containerID="75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089" Nov 03 03:45:15.977489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:15.977458 5275 scope.go:122] "RemoveContainer" containerID="09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54" Nov 03 03:45:16.188201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.188160 5275 generic.go:356] "Generic (PLEG): container finished" podID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerID="859017608fb22737a26a619196c9ebb3706e9fd5aabaa4ced60b21ee61e6fb9e" exitCode=60 Nov 03 03:45:16.188658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.188226 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerDied","Data":"859017608fb22737a26a619196c9ebb3706e9fd5aabaa4ced60b21ee61e6fb9e"} Nov 03 03:45:16.188658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.188291 5275 scope.go:122] "RemoveContainer" containerID="75fb70bc9002b96742275632c315258674b098f27ca6f04532e5c3313e89f089" Nov 03 03:45:16.188798 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.188725 5275 scope.go:122] "RemoveContainer" containerID="859017608fb22737a26a619196c9ebb3706e9fd5aabaa4ced60b21ee61e6fb9e" Nov 03 03:45:16.189139 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:16.189113 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc_odh-ai-gateway-infra(0694e9f5-4cd9-4deb-b3d9-03231046d9d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" Nov 03 03:45:16.977327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.977296 5275 scope.go:122] "RemoveContainer" containerID="0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d" Nov 03 03:45:16.977480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:16.977441 5275 scope.go:122] "RemoveContainer" containerID="ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d" Nov 03 03:45:17.197972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.197937 5275 generic.go:356] "Generic (PLEG): container finished" podID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerID="d062d7b3e7edd6f696f56837597e2d521707067f807c8c6a3758b6638ed16aab" exitCode=60 Nov 03 03:45:17.198388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.198011 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerDied","Data":"d062d7b3e7edd6f696f56837597e2d521707067f807c8c6a3758b6638ed16aab"} Nov 03 03:45:17.198388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.198069 5275 scope.go:122] "RemoveContainer" containerID="09c3351086a8b497150f88141b17a634b70e26eb06757d9f3adecc4cf2034e54" Nov 03 03:45:17.198524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.198507 5275 scope.go:122] "RemoveContainer" containerID="d062d7b3e7edd6f696f56837597e2d521707067f807c8c6a3758b6638ed16aab" Nov 03 03:45:17.198914 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:17.198882 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9_odh-ai-gateway-infra(f48fa82b-0be1-4b1b-9bf5-326b6ebd6464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" Nov 03 03:45:17.315181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.315147 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:17.509356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.509328 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:17.648389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.648343 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\") pod \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " Nov 03 03:45:17.648588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.648495 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\" (UniqueName: \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\") pod \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\" (UID: \"0694e9f5-4cd9-4deb-b3d9-03231046d9d0\") " Nov 03 03:45:17.648768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.648743 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca" pod "0694e9f5-4cd9-4deb-b3d9-03231046d9d0" (UID: "0694e9f5-4cd9-4deb-b3d9-03231046d9d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:17.649017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.648991 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:17.651537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.651501 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2" pod "0694e9f5-4cd9-4deb-b3d9-03231046d9d0" (UID: "0694e9f5-4cd9-4deb-b3d9-03231046d9d0"). InnerVolumeSpecName "kube-api-access-qnnz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:17.750614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.750530 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnz2\" (UniqueName: \"kubernetes.io/projected/0694e9f5-4cd9-4deb-b3d9-03231046d9d0-kube-api-access-qnnz2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:17.978336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.977865 5275 scope.go:122] "RemoveContainer" containerID="bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67" Nov 03 03:45:17.978336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.977978 5275 scope.go:122] "RemoveContainer" containerID="f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a" Nov 03 03:45:17.978598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:17.978369 5275 scope.go:122] "RemoveContainer" containerID="9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364" Nov 03 03:45:18.210147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.210117 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" Nov 03 03:45:18.210593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.210126 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc" event={"ID":"0694e9f5-4cd9-4deb-b3d9-03231046d9d0","Type":"ContainerDied","Data":"b47cd51c326c76db65b024f36cb19e687b5c73024478770e6eb7bac8abab0722"} Nov 03 03:45:18.210593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.210259 5275 scope.go:122] "RemoveContainer" containerID="859017608fb22737a26a619196c9ebb3706e9fd5aabaa4ced60b21ee61e6fb9e" Nov 03 03:45:18.212259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.212230 5275 generic.go:356] "Generic (PLEG): container finished" podID="6e55b5f7-e13d-405a-9813-378da3636323" containerID="438cc64f3aed8a79eaf31f5cb1c75c42547003b54667bf5c85377dd84cc31636" exitCode=60 Nov 03 03:45:18.212369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.212298 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerDied","Data":"438cc64f3aed8a79eaf31f5cb1c75c42547003b54667bf5c85377dd84cc31636"} Nov 03 03:45:18.212748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.212729 5275 scope.go:122] "RemoveContainer" containerID="438cc64f3aed8a79eaf31f5cb1c75c42547003b54667bf5c85377dd84cc31636" Nov 03 03:45:18.213082 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:18.213060 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp_odh-ai-gateway-infra(6e55b5f7-e13d-405a-9813-378da3636323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" podUID="6e55b5f7-e13d-405a-9813-378da3636323" Nov 03 03:45:18.214724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.214705 5275 generic.go:356] "Generic (PLEG): container finished" podID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerID="8da367c8357aa9034a26b6f44e6da743e24b0195f5f51bc84113fbd6eb064900" exitCode=60 Nov 03 03:45:18.214810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.214784 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerDied","Data":"8da367c8357aa9034a26b6f44e6da743e24b0195f5f51bc84113fbd6eb064900"} Nov 03 03:45:18.215158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.215144 5275 scope.go:122] "RemoveContainer" containerID="8da367c8357aa9034a26b6f44e6da743e24b0195f5f51bc84113fbd6eb064900" Nov 03 03:45:18.215400 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:18.215379 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9_odh-ai-gateway-infra(59137ac2-e3c6-490a-9566-114a9f2b1e8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" Nov 03 03:45:18.226976 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.226933 5275 scope.go:122] "RemoveContainer" containerID="0e55d7aa3d48b7d54719ab6f0c5fd0126198900b1b6fd7f3791e71ada098579d" Nov 03 03:45:18.243547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.243516 5275 scope.go:122] "RemoveContainer" containerID="ed39691eaf6e76b58a6e2579267bdfecc440a70ab75438bd5027f6535a1f5d0d" Nov 03 03:45:18.328783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.328751 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:18.376937 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.376906 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:18.384686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.384612 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b9c10a-29894625-22pnc"] Nov 03 03:45:18.551923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.551899 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:18.662740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.662689 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\" (UniqueName: \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\") pod \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " Nov 03 03:45:18.662930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.662765 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\") pod \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\" (UID: \"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464\") " Nov 03 03:45:18.663160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.663133 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca" pod "f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" (UID: "f48fa82b-0be1-4b1b-9bf5-326b6ebd6464"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:18.663370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.663353 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:18.665026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.665003 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g" pod "f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" (UID: "f48fa82b-0be1-4b1b-9bf5-326b6ebd6464"). InnerVolumeSpecName "kube-api-access-klx8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:18.764409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.764366 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klx8g\" (UniqueName: \"kubernetes.io/projected/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464-kube-api-access-klx8g\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:18.982379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.982341 5275 scope.go:122] "RemoveContainer" containerID="6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a" Nov 03 03:45:18.985925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:18.985904 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" path="/var/lib/kubelet/pods/0694e9f5-4cd9-4deb-b3d9-03231046d9d0/volumes" Nov 03 03:45:19.224508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.224475 5275 generic.go:356] "Generic (PLEG): container finished" podID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerID="525c7e47b5c236548b3ffda30088121566b323a4d858d1f18b982c0ecd076225" exitCode=60 Nov 03 03:45:19.224994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.224552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerDied","Data":"525c7e47b5c236548b3ffda30088121566b323a4d858d1f18b982c0ecd076225"} Nov 03 03:45:19.224994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.224628 5275 scope.go:122] "RemoveContainer" containerID="f761e25df07c4b9f83de1d9e3669dd0e37cb137552e895efbc7dec6bc6e2d93a" Nov 03 03:45:19.225129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.225109 5275 scope.go:122] "RemoveContainer" containerID="525c7e47b5c236548b3ffda30088121566b323a4d858d1f18b982c0ecd076225" Nov 03 03:45:19.225482 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:19.225453 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs_odh-ai-gateway-infra(2890d82d-69bd-4d36-a543-d9991ea53d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" Nov 03 03:45:19.226731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.226710 5275 generic.go:356] "Generic (PLEG): container finished" podID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerID="ea856f41d8ac278ed5b023c59fb0f395d8aa510e489e05cd826bb9dc2a625ab9" exitCode=60 Nov 03 03:45:19.226857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.226780 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerDied","Data":"ea856f41d8ac278ed5b023c59fb0f395d8aa510e489e05cd826bb9dc2a625ab9"} Nov 03 03:45:19.227161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.227143 5275 scope.go:122] "RemoveContainer" containerID="ea856f41d8ac278ed5b023c59fb0f395d8aa510e489e05cd826bb9dc2a625ab9" Nov 03 03:45:19.227451 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:19.227432 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp_odh-ai-gateway-infra(5c6db340-11fa-4f82-bfbe-901af3acb2d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" Nov 03 03:45:19.228239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.228225 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" Nov 03 03:45:19.228349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.228267 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9" event={"ID":"f48fa82b-0be1-4b1b-9bf5-326b6ebd6464","Type":"ContainerDied","Data":"d2cf7dedd18294dbb9c87ac5ea1a3991fcda2c9561b9d3808f8841f0abadb566"} Nov 03 03:45:19.235144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.235118 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerStarted","Data":"ce022d537825a945df9132b785e71f0ad0eb2d15924afe87f9b1cb332013dc1f"} Nov 03 03:45:19.242944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.242917 5275 scope.go:122] "RemoveContainer" containerID="bcc3525add25dc2fb81adad577ef1c27273b5a149646094c20cc222d46a62b67" Nov 03 03:45:19.260680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.260660 5275 scope.go:122] "RemoveContainer" containerID="d062d7b3e7edd6f696f56837597e2d521707067f807c8c6a3758b6638ed16aab" Nov 03 03:45:19.505303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.505216 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:19.621820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.621766 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:19.652771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.652740 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:19.690794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.690760 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:19.694877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.694818 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3480a2-29894625-6q5h9"] Nov 03 03:45:19.770275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.770247 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:19.780379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.780353 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\" (UniqueName: \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\") pod \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " Nov 03 03:45:19.780478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.780408 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\") pod \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\" (UID: \"59137ac2-e3c6-490a-9566-114a9f2b1e8d\") " Nov 03 03:45:19.780880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.780830 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca" pod "59137ac2-e3c6-490a-9566-114a9f2b1e8d" (UID: "59137ac2-e3c6-490a-9566-114a9f2b1e8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:19.782557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.782537 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2" pod "59137ac2-e3c6-490a-9566-114a9f2b1e8d" (UID: "59137ac2-e3c6-490a-9566-114a9f2b1e8d"). InnerVolumeSpecName "kube-api-access-b4vn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:19.882235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882199 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\" (UniqueName: \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\") pod \"6e55b5f7-e13d-405a-9813-378da3636323\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " Nov 03 03:45:19.882375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882353 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\") pod \"6e55b5f7-e13d-405a-9813-378da3636323\" (UID: \"6e55b5f7-e13d-405a-9813-378da3636323\") " Nov 03 03:45:19.882689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882662 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca" pod "6e55b5f7-e13d-405a-9813-378da3636323" (UID: "6e55b5f7-e13d-405a-9813-378da3636323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:19.882914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882897 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e55b5f7-e13d-405a-9813-378da3636323-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:19.882981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882919 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vn2\" (UniqueName: \"kubernetes.io/projected/59137ac2-e3c6-490a-9566-114a9f2b1e8d-kube-api-access-b4vn2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:19.882981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.882935 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59137ac2-e3c6-490a-9566-114a9f2b1e8d-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:19.884525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.884502 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k" pod "6e55b5f7-e13d-405a-9813-378da3636323" (UID: "6e55b5f7-e13d-405a-9813-378da3636323"). InnerVolumeSpecName "kube-api-access-dkl7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:19.984000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:19.983965 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkl7k\" (UniqueName: \"kubernetes.io/projected/6e55b5f7-e13d-405a-9813-378da3636323-kube-api-access-dkl7k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:20.243735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.243704 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" Nov 03 03:45:20.244246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.243707 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp" event={"ID":"6e55b5f7-e13d-405a-9813-378da3636323","Type":"ContainerDied","Data":"1f68e8b2c1c9f147685959cba9516a1de3a6b7dd54457f7ec1e22abd029848a3"} Nov 03 03:45:20.244246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.243891 5275 scope.go:122] "RemoveContainer" containerID="438cc64f3aed8a79eaf31f5cb1c75c42547003b54667bf5c85377dd84cc31636" Nov 03 03:45:20.246124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.246095 5275 generic.go:356] "Generic (PLEG): container finished" podID="6407137b-8c3d-4319-a853-a026612b6922" containerID="f6df453f06869822887bd2922e9f0790b18cb630e719eb6a6bf22a060bdadd37" exitCode=60 Nov 03 03:45:20.246240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.246122 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerDied","Data":"f6df453f06869822887bd2922e9f0790b18cb630e719eb6a6bf22a060bdadd37"} Nov 03 03:45:20.246535 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.246518 5275 scope.go:122] "RemoveContainer" containerID="f6df453f06869822887bd2922e9f0790b18cb630e719eb6a6bf22a060bdadd37" Nov 03 03:45:20.246836 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:20.246817 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5_odh-ai-gateway-infra(6407137b-8c3d-4319-a853-a026612b6922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" podUID="6407137b-8c3d-4319-a853-a026612b6922" Nov 03 03:45:20.247885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.247839 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" Nov 03 03:45:20.247982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.247878 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9" event={"ID":"59137ac2-e3c6-490a-9566-114a9f2b1e8d","Type":"ContainerDied","Data":"8da3df6da79fab63899922ece5f3abbfdc5e8b20980bc0700182804ffe64aade"} Nov 03 03:45:20.250191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.250172 5275 generic.go:356] "Generic (PLEG): container finished" podID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerID="ce022d537825a945df9132b785e71f0ad0eb2d15924afe87f9b1cb332013dc1f" exitCode=60 Nov 03 03:45:20.250298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.250285 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerDied","Data":"ce022d537825a945df9132b785e71f0ad0eb2d15924afe87f9b1cb332013dc1f"} Nov 03 03:45:20.250612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.250595 5275 scope.go:122] "RemoveContainer" containerID="ce022d537825a945df9132b785e71f0ad0eb2d15924afe87f9b1cb332013dc1f" Nov 03 03:45:20.250836 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:20.250818 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96_odh-ai-gateway-infra(4b497401-c25f-448f-a8ba-28cc0b3902c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" Nov 03 03:45:20.256203 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.256186 5275 scope.go:122] "RemoveContainer" containerID="6e19607a54ba3ff946925bf0ebb77a2dacb81c3fa30853f7a24caf26adfdab5a" Nov 03 03:45:20.270314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.270278 5275 scope.go:122] "RemoveContainer" containerID="8da367c8357aa9034a26b6f44e6da743e24b0195f5f51bc84113fbd6eb064900" Nov 03 03:45:20.279500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.279481 5275 scope.go:122] "RemoveContainer" containerID="9b91e8a52c04ba8339a8420b86217fc0123b1bd61e3ce630607249c8c7f80364" Nov 03 03:45:20.351436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.351357 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:20.466949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.466905 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:20.527162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.527131 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:20.536349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.536317 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a6b9ff-29894625-dxwn9"] Nov 03 03:45:20.540361 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.540335 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:20.608061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.607974 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:20.632817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.632790 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:20.701105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.701067 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:20.703626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.703595 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\") pod \"2890d82d-69bd-4d36-a543-d9991ea53d97\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " Nov 03 03:45:20.703795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.703663 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\") pod \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " Nov 03 03:45:20.703795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.703724 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\" (UniqueName: \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\") pod \"2890d82d-69bd-4d36-a543-d9991ea53d97\" (UID: \"2890d82d-69bd-4d36-a543-d9991ea53d97\") " Nov 03 03:45:20.703936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.703913 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\" (UniqueName: \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\") pod \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\" (UID: \"5c6db340-11fa-4f82-bfbe-901af3acb2d3\") " Nov 03 03:45:20.704079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.704054 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca" pod "2890d82d-69bd-4d36-a543-d9991ea53d97" (UID: "2890d82d-69bd-4d36-a543-d9991ea53d97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:20.704148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.704080 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca" pod "5c6db340-11fa-4f82-bfbe-901af3acb2d3" (UID: "5c6db340-11fa-4f82-bfbe-901af3acb2d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:20.704658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.704634 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6db340-11fa-4f82-bfbe-901af3acb2d3-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:20.704792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.704661 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2890d82d-69bd-4d36-a543-d9991ea53d97-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:20.706160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.706132 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-29dd3f-29894625-wxzkp"] Nov 03 03:45:20.706260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.706176 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t" pod "2890d82d-69bd-4d36-a543-d9991ea53d97" (UID: "2890d82d-69bd-4d36-a543-d9991ea53d97"). InnerVolumeSpecName "kube-api-access-6m28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:20.706260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.706206 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll" pod "5c6db340-11fa-4f82-bfbe-901af3acb2d3" (UID: "5c6db340-11fa-4f82-bfbe-901af3acb2d3"). InnerVolumeSpecName "kube-api-access-nt8ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:20.805971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.805935 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m28t\" (UniqueName: \"kubernetes.io/projected/2890d82d-69bd-4d36-a543-d9991ea53d97-kube-api-access-6m28t\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:20.805971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.805965 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8ll\" (UniqueName: \"kubernetes.io/projected/5c6db340-11fa-4f82-bfbe-901af3acb2d3-kube-api-access-nt8ll\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:20.987705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.987671 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" path="/var/lib/kubelet/pods/59137ac2-e3c6-490a-9566-114a9f2b1e8d/volumes" Nov 03 03:45:20.988215 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.988196 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e55b5f7-e13d-405a-9813-378da3636323" path="/var/lib/kubelet/pods/6e55b5f7-e13d-405a-9813-378da3636323/volumes" Nov 03 03:45:20.988577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:20.988564 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" path="/var/lib/kubelet/pods/f48fa82b-0be1-4b1b-9bf5-326b6ebd6464/volumes" Nov 03 03:45:21.269684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.269658 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" Nov 03 03:45:21.270092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.269663 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs" event={"ID":"2890d82d-69bd-4d36-a543-d9991ea53d97","Type":"ContainerDied","Data":"84a85fe7a2a4ccc3a7643ff57fa8547813564090bfdee7e6f202c164e17e3efc"} Nov 03 03:45:21.270092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.269804 5275 scope.go:122] "RemoveContainer" containerID="525c7e47b5c236548b3ffda30088121566b323a4d858d1f18b982c0ecd076225" Nov 03 03:45:21.271188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.271163 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" event={"ID":"5c6db340-11fa-4f82-bfbe-901af3acb2d3","Type":"ContainerDied","Data":"5337e52490af1b5fb58ef63555492f7a481059e4190dbc9f79cec5a496c22e84"} Nov 03 03:45:21.271280 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.271180 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp" Nov 03 03:45:21.282169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.282139 5275 scope.go:122] "RemoveContainer" containerID="ea856f41d8ac278ed5b023c59fb0f395d8aa510e489e05cd826bb9dc2a625ab9" Nov 03 03:45:21.429103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.429077 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:21.496613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.496579 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:21.506104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.506073 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894625-bz6fs"] Nov 03 03:45:21.514910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.514870 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\") pod \"4b497401-c25f-448f-a8ba-28cc0b3902c8\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " Nov 03 03:45:21.515068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.514985 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\" (UniqueName: \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\") pod \"4b497401-c25f-448f-a8ba-28cc0b3902c8\" (UID: \"4b497401-c25f-448f-a8ba-28cc0b3902c8\") " Nov 03 03:45:21.515180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.515154 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca" pod "4b497401-c25f-448f-a8ba-28cc0b3902c8" (UID: "4b497401-c25f-448f-a8ba-28cc0b3902c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:21.515614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.515592 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b497401-c25f-448f-a8ba-28cc0b3902c8-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:21.517219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.517194 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg" pod "4b497401-c25f-448f-a8ba-28cc0b3902c8" (UID: "4b497401-c25f-448f-a8ba-28cc0b3902c8"). InnerVolumeSpecName "kube-api-access-7bdfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:21.614651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.614613 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:21.616767 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.616738 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bdfg\" (UniqueName: \"kubernetes.io/projected/4b497401-c25f-448f-a8ba-28cc0b3902c8-kube-api-access-7bdfg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:21.619219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.619195 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-754aeb-29894625-xr6gp"] Nov 03 03:45:21.830827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:21.830730 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:22.002130 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.002099 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:22.133065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.132972 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\") pod \"6407137b-8c3d-4319-a853-a026612b6922\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " Nov 03 03:45:22.133244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.133088 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\" (UniqueName: \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\") pod \"6407137b-8c3d-4319-a853-a026612b6922\" (UID: \"6407137b-8c3d-4319-a853-a026612b6922\") " Nov 03 03:45:22.139272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.139229 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca" pod "6407137b-8c3d-4319-a853-a026612b6922" (UID: "6407137b-8c3d-4319-a853-a026612b6922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:22.142090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.142059 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv" pod "6407137b-8c3d-4319-a853-a026612b6922" (UID: "6407137b-8c3d-4319-a853-a026612b6922"). InnerVolumeSpecName "kube-api-access-hqvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:22.234526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.234495 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407137b-8c3d-4319-a853-a026612b6922-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:22.234526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.234523 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvcv\" (UniqueName: \"kubernetes.io/projected/6407137b-8c3d-4319-a853-a026612b6922-kube-api-access-hqvcv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:22.281485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.281443 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" event={"ID":"6407137b-8c3d-4319-a853-a026612b6922","Type":"ContainerDied","Data":"580b7708122a2b22c35c57c428b3c6ad480b9da5d72b6719f7ecb750e0ad9808"} Nov 03 03:45:22.281967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.281453 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5" Nov 03 03:45:22.281967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.281504 5275 scope.go:122] "RemoveContainer" containerID="f6df453f06869822887bd2922e9f0790b18cb630e719eb6a6bf22a060bdadd37" Nov 03 03:45:22.283086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.283063 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" Nov 03 03:45:22.283199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.283095 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96" event={"ID":"4b497401-c25f-448f-a8ba-28cc0b3902c8","Type":"ContainerDied","Data":"3b5ade924d144f151c6f45ad0e85774ac11079b15a03af83bd0bdda92518ab88"} Nov 03 03:45:22.292216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.292196 5275 scope.go:122] "RemoveContainer" containerID="ce022d537825a945df9132b785e71f0ad0eb2d15924afe87f9b1cb332013dc1f" Nov 03 03:45:22.448760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.448644 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:22.451522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.451498 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894625-w7h96"] Nov 03 03:45:22.559021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.558979 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:22.566839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.566802 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b755ee-29894625-knlm5"] Nov 03 03:45:22.985809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.985773 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" path="/var/lib/kubelet/pods/2890d82d-69bd-4d36-a543-d9991ea53d97/volumes" Nov 03 03:45:22.986249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.986233 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" path="/var/lib/kubelet/pods/4b497401-c25f-448f-a8ba-28cc0b3902c8/volumes" Nov 03 03:45:22.986605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.986591 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" path="/var/lib/kubelet/pods/5c6db340-11fa-4f82-bfbe-901af3acb2d3/volumes" Nov 03 03:45:22.987323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:22.987304 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6407137b-8c3d-4319-a853-a026612b6922" path="/var/lib/kubelet/pods/6407137b-8c3d-4319-a853-a026612b6922/volumes" Nov 03 03:45:25.460682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.460644 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z"] Nov 03 03:45:25.461638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461605 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cad05144-5d31-4641-b471-395f7377e6fc" containerName="main" Nov 03 03:45:25.461638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461632 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad05144-5d31-4641-b471-395f7377e6fc" containerName="main" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461645 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461652 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461672 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461679 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461694 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461700 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461711 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461718 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461733 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461739 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461758 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461765 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461776 5275 cpu_manager.go:414] "containerMap: removing container" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461782 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461795 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.461795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461802 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461816 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461822 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461833 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461840 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461867 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461873 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461886 5275 cpu_manager.go:414] "containerMap: removing container" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461901 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461929 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461935 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461951 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461957 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461971 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461977 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461989 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.461995 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462005 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462011 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462020 5275 cpu_manager.go:414] "containerMap: removing container" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462026 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462050 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462057 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462204 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462218 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462227 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462235 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462243 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad05144-5d31-4641-b471-395f7377e6fc" containerName="main" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462254 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462261 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462270 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462280 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462292 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462301 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462310 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462319 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462326 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462334 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462343 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462353 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462362 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462373 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462382 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:45:25.462589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.462391 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:45:25.467782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.467761 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.525729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.516087 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z"] Nov 03 03:45:25.560729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.560561 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:45:25.560729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.560615 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:45:25.560729 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.560648 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:45:25.569234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.569187 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:45:25.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.577711 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh29\" (UniqueName: \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.577966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.577945 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.578024 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.578012 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.679177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.679122 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.679389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.679238 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.679389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.679369 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh29\" (UniqueName: \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.680451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.680422 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.682463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.682432 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.741290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.741210 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh29\" (UniqueName: \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\") pod \"maas-api-revoke-sub-7a22db0d-x777z\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:25.784350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:25.784310 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:26.164534 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:26.164500 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z"] Nov 03 03:45:26.164883 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:45:26.164840 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c30a4c_5949_4cb6_9e9e_245f5ddd93f2.slice/crio-72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd WatchSource:0}: Error finding container 72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd: Status 404 returned error can't find the container with id 72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd Nov 03 03:45:26.322623 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:26.322587 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" event={"ID":"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2","Type":"ContainerStarted","Data":"72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd"} Nov 03 03:45:27.331836 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:27.331740 5275 generic.go:356] "Generic (PLEG): container finished" podID="27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" containerID="230e2a375a6d347a9254496bbc5138acc36adea0be194b9ddeba50f1648dc97d" exitCode=0 Nov 03 03:45:27.332271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:27.331827 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" event={"ID":"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2","Type":"ContainerDied","Data":"230e2a375a6d347a9254496bbc5138acc36adea0be194b9ddeba50f1648dc97d"} Nov 03 03:45:28.519002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.518974 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:28.629896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.629782 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\") pod \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " Nov 03 03:45:28.629896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.629889 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\" (UniqueName: \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\") pod \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " Nov 03 03:45:28.630092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.630073 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\") pod \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\" (UID: \"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2\") " Nov 03 03:45:28.630205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.630183 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca" pod "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" (UID: "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:28.630521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.630493 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:28.632386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.632354 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29" pod "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" (UID: "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2"). InnerVolumeSpecName "kube-api-access-wkh29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:28.632513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.632433 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca" pod "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" (UID: "27c30a4c-5949-4cb6-9e9e-245f5ddd93f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:28.731745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.731695 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:28.731745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.731729 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkh29\" (UniqueName: \"kubernetes.io/projected/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2-kube-api-access-wkh29\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:28.780408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.780373 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:45:28.780608 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:28.780586 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" containerID="cri-o://5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4" gracePeriod=30 Nov 03 03:45:29.351291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:29.351248 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" event={"ID":"27c30a4c-5949-4cb6-9e9e-245f5ddd93f2","Type":"ContainerDied","Data":"72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd"} Nov 03 03:45:29.351628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:29.351312 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72dc769116005c0ea31b7b0bfb1d66d04d8a3734e0d6cdcfed95307a136978bd" Nov 03 03:45:29.351628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:29.351379 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z" Nov 03 03:45:32.330100 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:32.330061 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:45:34.111545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.111518 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:45:34.235500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235464 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.235500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235506 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.235737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235533 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.235737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235623 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.235885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235741 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.235885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235824 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:34.236012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235912 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.236012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235940 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:34.236012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.235981 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.236164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236014 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.236164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236099 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\") pod \"75f0a28b-af55-4874-a30f-c4452d75f2a5\" (UID: \"75f0a28b-af55-4874-a30f-c4452d75f2a5\") " Nov 03 03:45:34.236750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236499 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:34.236750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236598 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:34.236750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236698 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:34.237012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236825 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0a28b-af55-4874-a30f-c4452d75f2a5-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.237012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236866 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.237012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236882 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.237012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236897 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.237012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.236910 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.238250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.238185 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:34.238646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.238627 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "kube-api-access-zwf8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:34.238646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.238629 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:34.238776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.238745 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo" pod "75f0a28b-af55-4874-a30f-c4452d75f2a5" (UID: "75f0a28b-af55-4874-a30f-c4452d75f2a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:45:34.337687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.337603 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.337687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.337633 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwf8b\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-kube-api-access-zwf8b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.337687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.337644 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.337687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.337653 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0a28b-af55-4874-a30f-c4452d75f2a5-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:34.416357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.416323 5275 generic.go:356] "Generic (PLEG): container finished" podID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerID="5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4" exitCode=0 Nov 03 03:45:34.416560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.416401 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" Nov 03 03:45:34.416560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.416415 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerDied","Data":"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4"} Nov 03 03:45:34.416560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.416512 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm" event={"ID":"75f0a28b-af55-4874-a30f-c4452d75f2a5","Type":"ContainerDied","Data":"16d6efbbd6302a745d83afc9839929e23dc69c2c23e65a51734d3800745c2e04"} Nov 03 03:45:34.416560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.416544 5275 scope.go:122] "RemoveContainer" containerID="5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4" Nov 03 03:45:34.429477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.429448 5275 scope.go:122] "RemoveContainer" containerID="aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0" Nov 03 03:45:34.439512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.439487 5275 scope.go:122] "RemoveContainer" containerID="5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4" Nov 03 03:45:34.439876 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:34.439819 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4\": container with ID starting with 5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4 not found: ID does not exist" containerID="5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4" Nov 03 03:45:34.439935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.439868 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4"} err="failed to get container status \"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4\": rpc error: code = NotFound desc = could not find container \"5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4\": container with ID starting with 5224b11bab44b450bfb5e64b83b1b8a034a7bb242620cdebc5b115c5e9feddb4 not found: ID does not exist" Nov 03 03:45:34.439935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.439889 5275 scope.go:122] "RemoveContainer" containerID="aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0" Nov 03 03:45:34.440148 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:45:34.440131 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0\": container with ID starting with aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0 not found: ID does not exist" containerID="aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0" Nov 03 03:45:34.440189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.440155 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0"} err="failed to get container status \"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0\": rpc error: code = NotFound desc = could not find container \"aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0\": container with ID starting with aeda0b4a3fc6c0fe1927becf0a9743f5373a24de085720009a11ae142bda77d0 not found: ID does not exist" Nov 03 03:45:34.587015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.586980 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:45:34.592268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.592147 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-p8xpm"] Nov 03 03:45:34.992039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:34.991955 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" path="/var/lib/kubelet/pods/75f0a28b-af55-4874-a30f-c4452d75f2a5/volumes" Nov 03 03:45:39.463284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.463249 5275 generic.go:356] "Generic (PLEG): container finished" podID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerID="d11348047229602a94be7df2844e3bd4b3a0abb96a4cc18b2ba814758edf652f" exitCode=0 Nov 03 03:45:39.463777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.463334 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerDied","Data":"d11348047229602a94be7df2844e3bd4b3a0abb96a4cc18b2ba814758edf652f"} Nov 03 03:45:39.463935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.463917 5275 scope.go:122] "RemoveContainer" containerID="d11348047229602a94be7df2844e3bd4b3a0abb96a4cc18b2ba814758edf652f" Nov 03 03:45:39.465822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.465779 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:45:39.465974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.465888 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:45:39.465974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:39.465941 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:45:40.331490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:40.331453 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:45:40.498234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:40.498054 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerStarted","Data":"5b97cce412124a90c63c0960f0dfa2234c10d381163e9ace6fe508db79082dbc"} Nov 03 03:45:41.409289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:41.409253 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:45:41.410576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:41.410548 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 03:45:41.410714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:41.410601 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 03:45:42.408957 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:42.408921 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 03:45:42.409480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:42.409002 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 03:45:43.409465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:43.409428 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 03:45:43.409900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:43.409502 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 03:45:44.409010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:44.408973 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 03:45:44.409199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:44.409040 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 03:45:45.409696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:45.409653 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 03:45:45.410179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:45.409722 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 03:45:46.413997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:46.413967 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:45:46.414505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:46.414486 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:45:46.415310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:46.415290 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:45:55.641407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:55.641370 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:45:55.641978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:45:55.641594 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" containerID="cri-o://006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e" gracePeriod=30 Nov 03 03:46:00.926264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:00.926198 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:46:01.039239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039200 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039305 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039351 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039385 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039422 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039443 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:01.039698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039532 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039597 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039631 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.039698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039695 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\") pod \"e45d272d-573e-47a1-a7a2-e596dfb41bef\" (UID: \"e45d272d-573e-47a1-a7a2-e596dfb41bef\") " Nov 03 03:46:01.040008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039594 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:01.040008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039700 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:01.040008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.039818 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:01.040179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040125 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:01.040936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040908 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.041061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040939 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45d272d-573e-47a1-a7a2-e596dfb41bef-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.041061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040954 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.041061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040968 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.041061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.040984 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.042391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.042355 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:01.042391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.042371 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:46:01.042525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.042390 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "kube-api-access-tvwd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:01.042525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.042431 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token" pod "e45d272d-573e-47a1-a7a2-e596dfb41bef" (UID: "e45d272d-573e-47a1-a7a2-e596dfb41bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:01.142624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.142586 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwd8\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-kube-api-access-tvwd8\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.142624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.142619 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.142624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.142630 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.142889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.142638 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45d272d-573e-47a1-a7a2-e596dfb41bef-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:01.680564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.680530 5275 generic.go:356] "Generic (PLEG): container finished" podID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerID="006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e" exitCode=0 Nov 03 03:46:01.680782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.680611 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" Nov 03 03:46:01.680782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.680616 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerDied","Data":"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e"} Nov 03 03:46:01.680782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.680691 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w" event={"ID":"e45d272d-573e-47a1-a7a2-e596dfb41bef","Type":"ContainerDied","Data":"9fbbee0bbed0e8fe69972dd3d8a5148acb9d930fc4f3904d43082009d50ec641"} Nov 03 03:46:01.680782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.680724 5275 scope.go:122] "RemoveContainer" containerID="006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e" Nov 03 03:46:01.699470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.699436 5275 scope.go:122] "RemoveContainer" containerID="a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a" Nov 03 03:46:01.709775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.709749 5275 scope.go:122] "RemoveContainer" containerID="006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e" Nov 03 03:46:01.710218 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:01.710197 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e\": container with ID starting with 006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e not found: ID does not exist" containerID="006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e" Nov 03 03:46:01.710323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.710231 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e"} err="failed to get container status \"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e\": rpc error: code = NotFound desc = could not find container \"006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e\": container with ID starting with 006cff1b71c1e2332509bf31163ebfe05bf66e25bed0685e843cdde7c502393e not found: ID does not exist" Nov 03 03:46:01.710323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.710257 5275 scope.go:122] "RemoveContainer" containerID="a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a" Nov 03 03:46:01.710619 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:01.710601 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a\": container with ID starting with a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a not found: ID does not exist" containerID="a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a" Nov 03 03:46:01.710663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.710628 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a"} err="failed to get container status \"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a\": rpc error: code = NotFound desc = could not find container \"a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a\": container with ID starting with a93df445902ca950160ca00faa3ec113b448edb91f667aa6d5aca7c114a8f24a not found: ID does not exist" Nov 03 03:46:01.848270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.848224 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:46:01.898142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:01.898098 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b9ff-openshift-default-86d56f7875-7tr8w"] Nov 03 03:46:02.329871 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:02.329820 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:46:02.988220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:02.988182 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" path="/var/lib/kubelet/pods/e45d272d-573e-47a1-a7a2-e596dfb41bef/volumes" Nov 03 03:46:02.988653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:02.988634 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:46:03.096470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:03.096434 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:46:03.808167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:03.808132 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:46:03.808930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:03.808898 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" podUID="a09fb47d-651c-4555-b026-1425f65b4148" containerName="payload-processing" containerID="cri-o://d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9" gracePeriod=30 Nov 03 03:46:04.118353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.118326 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:46:04.294437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.294404 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\") pod \"a09fb47d-651c-4555-b026-1425f65b4148\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " Nov 03 03:46:04.294644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.294469 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\" (UniqueName: \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\") pod \"a09fb47d-651c-4555-b026-1425f65b4148\" (UID: \"a09fb47d-651c-4555-b026-1425f65b4148\") " Nov 03 03:46:04.294863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.294824 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume" pod "a09fb47d-651c-4555-b026-1425f65b4148" (UID: "a09fb47d-651c-4555-b026-1425f65b4148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:04.296683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.296656 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b" pod "a09fb47d-651c-4555-b026-1425f65b4148" (UID: "a09fb47d-651c-4555-b026-1425f65b4148"). InnerVolumeSpecName "kube-api-access-9bc4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:04.395875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.395747 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09fb47d-651c-4555-b026-1425f65b4148-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.395875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.395777 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bc4b\" (UniqueName: \"kubernetes.io/projected/a09fb47d-651c-4555-b026-1425f65b4148-kube-api-access-9bc4b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.709472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.709383 5275 generic.go:356] "Generic (PLEG): container finished" podID="a09fb47d-651c-4555-b026-1425f65b4148" containerID="d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9" exitCode=0 Nov 03 03:46:04.709472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.709439 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" Nov 03 03:46:04.709689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.709473 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" event={"ID":"a09fb47d-651c-4555-b026-1425f65b4148","Type":"ContainerDied","Data":"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9"} Nov 03 03:46:04.709689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.709522 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj" event={"ID":"a09fb47d-651c-4555-b026-1425f65b4148","Type":"ContainerDied","Data":"e3d5d554ca093439067f3c8cff34d3bdff52ec0fd9afd06569a576f5b6600f90"} Nov 03 03:46:04.709689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.709544 5275 scope.go:122] "RemoveContainer" containerID="d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9" Nov 03 03:46:04.723740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.723717 5275 scope.go:122] "RemoveContainer" containerID="d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9" Nov 03 03:46:04.724230 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:04.724196 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9\": container with ID starting with d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9 not found: ID does not exist" containerID="d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9" Nov 03 03:46:04.724376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.724242 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9"} err="failed to get container status \"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9\": rpc error: code = NotFound desc = could not find container \"d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9\": container with ID starting with d6cf6c9ed46d1aafae2a85e7bfc313a2f768764910599a2d349e9ba7f920ebf9 not found: ID does not exist" Nov 03 03:46:04.883073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.883037 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:46:04.886092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.886067 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b9c10a-6f768764f6-w6ncj"] Nov 03 03:46:04.985477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:04.985399 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09fb47d-651c-4555-b026-1425f65b4148" path="/var/lib/kubelet/pods/a09fb47d-651c-4555-b026-1425f65b4148/volumes" Nov 03 03:46:07.989530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:07.989445 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" podUID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" containerName="payload-processing" containerID="cri-o://c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e" gracePeriod=25 Nov 03 03:46:08.097881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.096977 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" podUID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" containerName="payload-pre-processing" containerID="cri-o://27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4" gracePeriod=25 Nov 03 03:46:08.404871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.398239 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:46:08.460000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.459963 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\" (UniqueName: \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\") pod \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " Nov 03 03:46:08.460223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.460152 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\") pod \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\" (UID: \"2a33a003-b652-4b11-bf56-f8c8b254d3b5\") " Nov 03 03:46:08.460759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.460721 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume" pod "2a33a003-b652-4b11-bf56-f8c8b254d3b5" (UID: "2a33a003-b652-4b11-bf56-f8c8b254d3b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:08.465783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.465740 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj" pod "2a33a003-b652-4b11-bf56-f8c8b254d3b5" (UID: "2a33a003-b652-4b11-bf56-f8c8b254d3b5"). InnerVolumeSpecName "kube-api-access-kwctj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:08.497645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.497587 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.561262 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\") pod \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.561388 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\" (UniqueName: \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\") pod \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\" (UID: \"f89d22e0-bb48-403c-abdc-9fc2bbce046b\") " Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.561722 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume" pod "f89d22e0-bb48-403c-abdc-9fc2bbce046b" (UID: "f89d22e0-bb48-403c-abdc-9fc2bbce046b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.562097 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a33a003-b652-4b11-bf56-f8c8b254d3b5-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.562120 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwctj\" (UniqueName: \"kubernetes.io/projected/2a33a003-b652-4b11-bf56-f8c8b254d3b5-kube-api-access-kwctj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:08.562134 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.562135 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89d22e0-bb48-403c-abdc-9fc2bbce046b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:08.565265 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.565224 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t" pod "f89d22e0-bb48-403c-abdc-9fc2bbce046b" (UID: "f89d22e0-bb48-403c-abdc-9fc2bbce046b"). InnerVolumeSpecName "kube-api-access-7x49t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:08.663436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.663392 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x49t\" (UniqueName: \"kubernetes.io/projected/f89d22e0-bb48-403c-abdc-9fc2bbce046b-kube-api-access-7x49t\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:08.742622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.742584 5275 generic.go:356] "Generic (PLEG): container finished" podID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" containerID="c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e" exitCode=0 Nov 03 03:46:08.742824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.742633 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" Nov 03 03:46:08.742824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.742658 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" event={"ID":"2a33a003-b652-4b11-bf56-f8c8b254d3b5","Type":"ContainerDied","Data":"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e"} Nov 03 03:46:08.742824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.742713 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6" event={"ID":"2a33a003-b652-4b11-bf56-f8c8b254d3b5","Type":"ContainerDied","Data":"68f4dcbfb759041b08463169b6cd6839f575cb617b0f1fff57551d612a318c58"} Nov 03 03:46:08.742824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.742735 5275 scope.go:122] "RemoveContainer" containerID="c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e" Nov 03 03:46:08.745161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.745133 5275 generic.go:356] "Generic (PLEG): container finished" podID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" containerID="27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4" exitCode=0 Nov 03 03:46:08.745283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.745204 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" Nov 03 03:46:08.745283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.745201 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" event={"ID":"f89d22e0-bb48-403c-abdc-9fc2bbce046b","Type":"ContainerDied","Data":"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4"} Nov 03 03:46:08.745369 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.745333 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5" event={"ID":"f89d22e0-bb48-403c-abdc-9fc2bbce046b","Type":"ContainerDied","Data":"75de59b08a320d2bfb1985334bad81719ad5e97dfdc388cb8bb1b9a2a10da7e9"} Nov 03 03:46:08.756802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.756782 5275 scope.go:122] "RemoveContainer" containerID="c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e" Nov 03 03:46:08.757236 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:08.757210 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e\": container with ID starting with c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e not found: ID does not exist" containerID="c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e" Nov 03 03:46:08.757346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.757256 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e"} err="failed to get container status \"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e\": rpc error: code = NotFound desc = could not find container \"c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e\": container with ID starting with c207fcb688e2568a2bb292b5529c51b46a80c3f6e1abec6eb9566eec26e7b48e not found: ID does not exist" Nov 03 03:46:08.757346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.757282 5275 scope.go:122] "RemoveContainer" containerID="27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4" Nov 03 03:46:08.768970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.768925 5275 scope.go:122] "RemoveContainer" containerID="27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4" Nov 03 03:46:08.769364 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:08.769340 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4\": container with ID starting with 27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4 not found: ID does not exist" containerID="27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4" Nov 03 03:46:08.769483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.769371 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4"} err="failed to get container status \"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4\": rpc error: code = NotFound desc = could not find container \"27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4\": container with ID starting with 27cbafca42a42378e73ad34a019012a49ba34121c2bd4af530119811fd70b9a4 not found: ID does not exist" Nov 03 03:46:08.907096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.907057 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:46:08.909877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.909828 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3480a2-d6f66dbc5-pmkw5"] Nov 03 03:46:08.985618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:08.985584 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" path="/var/lib/kubelet/pods/f89d22e0-bb48-403c-abdc-9fc2bbce046b/volumes" Nov 03 03:46:09.021089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:09.021056 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:46:09.026641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:09.026610 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3480a2-774566f56f-fj4r6"] Nov 03 03:46:10.686163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:10.686120 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:46:10.686601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:10.686380 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" containerID="cri-o://6043c036127fbc11d5dfa1a31777e558aa0b30d39b20b0000107552345114605" gracePeriod=30 Nov 03 03:46:10.995379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:10.995292 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" path="/var/lib/kubelet/pods/2a33a003-b652-4b11-bf56-f8c8b254d3b5/volumes" Nov 03 03:46:12.691315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.691274 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:46:12.692137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692115 5275 cpu_manager.go:414] "containerMap: removing container" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692139 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692165 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" containerName="payload-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692172 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" containerName="payload-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692181 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" containerName="payload-pre-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692189 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" containerName="payload-pre-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692204 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a09fb47d-651c-4555-b026-1425f65b4148" containerName="payload-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692211 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09fb47d-651c-4555-b026-1425f65b4148" containerName="payload-processing" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692223 5275 cpu_manager.go:414] "containerMap: removing container" podUID="27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" containerName="revoke-keys" Nov 03 03:46:12.692238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692230 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" containerName="revoke-keys" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692249 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692255 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48fa82b-0be1-4b1b-9bf5-326b6ebd6464" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692264 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692271 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6db340-11fa-4f82-bfbe-901af3acb2d3" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692293 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692300 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e55b5f7-e13d-405a-9813-378da3636323" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692321 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692328 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692338 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692346 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6407137b-8c3d-4319-a853-a026612b6922" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692381 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692388 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692513 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692526 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692536 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0694e9f5-4cd9-4deb-b3d9-03231046d9d0" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692546 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a33a003-b652-4b11-bf56-f8c8b254d3b5" containerName="payload-processing" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692554 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89d22e0-bb48-403c-abdc-9fc2bbce046b" containerName="payload-pre-processing" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692566 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09fb47d-651c-4555-b026-1425f65b4148" containerName="payload-processing" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692575 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" containerName="revoke-keys" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692587 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="59137ac2-e3c6-490a-9566-114a9f2b1e8d" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692600 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692608 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2890d82d-69bd-4d36-a543-d9991ea53d97" containerName="cleanup" Nov 03 03:46:12.692696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.692620 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b497401-c25f-448f-a8ba-28cc0b3902c8" containerName="cleanup" Nov 03 03:46:12.697769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.697742 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.778101 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.778063 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5ec4ce-openshift-default-dockercfg-2q5fn\"" Nov 03 03:46:12.817028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.816987 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823087 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m2f\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823149 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823195 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823226 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823403 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823468 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823499 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823542 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.823612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.823569 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924555 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924594 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924617 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924676 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m2f\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924694 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924731 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.924914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924760 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924934 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.924997 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.925137 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.925208 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.925330 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925583 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.925439 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.925748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.925628 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.927288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.927263 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.927400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.927328 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.987704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.987635 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:12.987948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:12.987922 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m2f\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\") pod \"e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:13.011358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.011326 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:13.451115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.451086 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:46:13.453365 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:46:13.453338 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc323c990_67af_4a0b_9b89_dc5418fc7648.slice/crio-b848308f3f2170a12a02dd0a7fa93b771744a3f16db89c1e5bbb3e1b1439847b WatchSource:0}: Error finding container b848308f3f2170a12a02dd0a7fa93b771744a3f16db89c1e5bbb3e1b1439847b: Status 404 returned error can't find the container with id b848308f3f2170a12a02dd0a7fa93b771744a3f16db89c1e5bbb3e1b1439847b Nov 03 03:46:13.456510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.456471 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:13.456602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.456559 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:13.456640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.456603 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:13.801792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.801754 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerStarted","Data":"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b"} Nov 03 03:46:13.802250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.801819 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerStarted","Data":"b848308f3f2170a12a02dd0a7fa93b771744a3f16db89c1e5bbb3e1b1439847b"} Nov 03 03:46:13.925261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:13.925205 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podStartSLOduration=1.925189249 podStartE2EDuration="1.925189249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:13.921259236 +0000 UTC m=+2665.073947657" watchObservedRunningTime="2026-11-03 03:46:13.925189249 +0000 UTC m=+2665.077877670" Nov 03 03:46:14.012401 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:14.012361 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:14.018149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:14.018117 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:14.808228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:14.808199 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:14.809186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:14.809171 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:46:15.817340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:15.817310 5275 generic.go:356] "Generic (PLEG): container finished" podID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerID="6043c036127fbc11d5dfa1a31777e558aa0b30d39b20b0000107552345114605" exitCode=0 Nov 03 03:46:15.817809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:15.817378 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerDied","Data":"6043c036127fbc11d5dfa1a31777e558aa0b30d39b20b0000107552345114605"} Nov 03 03:46:15.817809 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:15.817439 5275 scope.go:122] "RemoveContainer" containerID="471e7492ac7a70b8daf3b6abf727aaa26a771be9898c144cd41e283ca0dbeed3" Nov 03 03:46:15.988316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:15.988288 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:46:16.064751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.064709 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.064984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.064806 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065012 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065047 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065105 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065131 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:16.065174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065160 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065231 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065280 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\") pod \"34664fbe-649d-4092-9e9b-c9ca6783ed70\" (UID: \"34664fbe-649d-4092-9e9b-c9ca6783ed70\") " Nov 03 03:46:16.065478 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.065436 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:16.066209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.066043 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34664fbe-649d-4092-9e9b-c9ca6783ed70-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.066209 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.066070 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.066577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.066308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:16.066577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.066483 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:16.066781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.066749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:16.067617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.067581 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:16.067929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.067909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "kube-api-access-mv99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:16.068164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.068137 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:46:16.068555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.068526 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token" pod "34664fbe-649d-4092-9e9b-c9ca6783ed70" (UID: "34664fbe-649d-4092-9e9b-c9ca6783ed70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:16.167638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167608 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167636 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167646 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167657 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv99r\" (UniqueName: \"kubernetes.io/projected/34664fbe-649d-4092-9e9b-c9ca6783ed70-kube-api-access-mv99r\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167667 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167675 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34664fbe-649d-4092-9e9b-c9ca6783ed70-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.167906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.167685 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34664fbe-649d-4092-9e9b-c9ca6783ed70-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:16.826468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.826436 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" Nov 03 03:46:16.827007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.826482 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm" event={"ID":"34664fbe-649d-4092-9e9b-c9ca6783ed70","Type":"ContainerDied","Data":"5df1a33b64321bddc55ea78eab64cc2e46ce593fec1f8715082e57fe9f695f27"} Nov 03 03:46:16.827007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:16.826551 5275 scope.go:122] "RemoveContainer" containerID="6043c036127fbc11d5dfa1a31777e558aa0b30d39b20b0000107552345114605" Nov 03 03:46:17.042476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:17.042407 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:46:17.091161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:17.091079 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b9c10a-openshift-default-5989559c77-5nwtm"] Nov 03 03:46:18.984345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:18.984310 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" path="/var/lib/kubelet/pods/34664fbe-649d-4092-9e9b-c9ca6783ed70/volumes" Nov 03 03:46:20.523387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.523353 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:46:20.534947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.534918 5275 cpu_manager.go:414] "containerMap: removing container" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:20.535223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535200 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:20.535441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535429 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:20.535545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535534 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45d272d-573e-47a1-a7a2-e596dfb41bef" containerName="istio-proxy" Nov 03 03:46:20.535662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535652 5275 cpu_manager.go:414] "containerMap: removing container" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.535762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535751 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.535892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535880 5275 cpu_manager.go:414] "containerMap: removing container" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.536005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.535993 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.536313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.536293 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.536418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.536408 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f0a28b-af55-4874-a30f-c4452d75f2a5" containerName="istio-proxy" Nov 03 03:46:20.536500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.536491 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="34664fbe-649d-4092-9e9b-c9ca6783ed70" containerName="istio-proxy" Nov 03 03:46:20.553868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.547146 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.581080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.581047 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:46:20.603229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.603204 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5ec4ce-dockercfg-2mst7\"" Nov 03 03:46:20.603862 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.603819 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5ec4ce\"" Nov 03 03:46:20.623978 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.623935 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nzr\" (UniqueName: \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.624195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.624055 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.680017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.679981 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:46:20.684082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.684057 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.725414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.725366 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.725607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.725522 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nzr\" (UniqueName: \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.726444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.726414 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.742429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.742394 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:46:20.786825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.786751 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nzr\" (UniqueName: \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\") pod \"payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.826412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.826373 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.826613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.826595 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbk4k\" (UniqueName: \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.895248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.895209 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:20.927446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.927396 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbk4k\" (UniqueName: \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.927637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.927494 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.928312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.928290 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:20.991213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:20.991176 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbk4k\" (UniqueName: \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\") pod \"payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:21.004294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.004266 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:21.283796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.283766 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:46:21.284912 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:46:21.284880 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e25e042_7773_4ca9_877b_3d6622cbce12.slice/crio-785222f6f0e33f58a13dd1bbdc4128c42199fc1394f427cd064240eb4a91f58e WatchSource:0}: Error finding container 785222f6f0e33f58a13dd1bbdc4128c42199fc1394f427cd064240eb4a91f58e: Status 404 returned error can't find the container with id 785222f6f0e33f58a13dd1bbdc4128c42199fc1394f427cd064240eb4a91f58e Nov 03 03:46:21.386012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.385987 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:46:21.387871 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:46:21.387815 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4897ddb5_3587_4549_b44d_aa7799396dbb.slice/crio-eb3dcf6a98fbd87c8ddad2b6cf18af3c9fc5969c29fb7144601d6d32909be227 WatchSource:0}: Error finding container eb3dcf6a98fbd87c8ddad2b6cf18af3c9fc5969c29fb7144601d6d32909be227: Status 404 returned error can't find the container with id eb3dcf6a98fbd87c8ddad2b6cf18af3c9fc5969c29fb7144601d6d32909be227 Nov 03 03:46:21.865128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.865088 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" event={"ID":"8e25e042-7773-4ca9-877b-3d6622cbce12","Type":"ContainerStarted","Data":"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2"} Nov 03 03:46:21.865593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.865157 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" event={"ID":"8e25e042-7773-4ca9-877b-3d6622cbce12","Type":"ContainerStarted","Data":"785222f6f0e33f58a13dd1bbdc4128c42199fc1394f427cd064240eb4a91f58e"} Nov 03 03:46:21.865593 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.865308 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:21.866860 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.866820 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" event={"ID":"4897ddb5-3587-4549-b44d-aa7799396dbb","Type":"ContainerStarted","Data":"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a"} Nov 03 03:46:21.866990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.866896 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" event={"ID":"4897ddb5-3587-4549-b44d-aa7799396dbb","Type":"ContainerStarted","Data":"eb3dcf6a98fbd87c8ddad2b6cf18af3c9fc5969c29fb7144601d6d32909be227"} Nov 03 03:46:21.866990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.866927 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:21.983258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:21.983205 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" podStartSLOduration=1.983190149 podStartE2EDuration="1.983190149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:21.979539406 +0000 UTC m=+2673.132227828" watchObservedRunningTime="2026-11-03 03:46:21.983190149 +0000 UTC m=+2673.135878593" Nov 03 03:46:22.106673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:22.106578 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" podStartSLOduration=2.106555778 podStartE2EDuration="2.106555778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:22.104486908 +0000 UTC m=+2673.257175344" watchObservedRunningTime="2026-11-03 03:46:22.106555778 +0000 UTC m=+2673.259244200" Nov 03 03:46:27.571439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.571393 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:46:27.576353 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.576327 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.634867 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.634263 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e84ee2-openshift-default-dockercfg-ng9mg\"" Nov 03 03:46:27.691663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.691600 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:46:27.711925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.711890 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.711984 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712055 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712151 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712206 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712236 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpm8l\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712314 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712416 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712403 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.712485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.712472 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.813783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.813745 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.813783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.813803 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.813829 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.813892 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpm8l\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.813919 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814020 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814057 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814137 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814135 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814164 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814271 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814429 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814677 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814696 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.814775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.814704 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.816709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.816691 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.816877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.816839 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.874951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.874866 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.875118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.875055 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpm8l\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\") pod \"e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:27.947463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:27.947412 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:28.392559 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:46:28.392516 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c821073_1948_4520_98e5_ca080b896b1a.slice/crio-63f09b2cb901d39afeea39f9a52088865a0d056b17e36a8e6a79fa1157b4e7ae WatchSource:0}: Error finding container 63f09b2cb901d39afeea39f9a52088865a0d056b17e36a8e6a79fa1157b4e7ae: Status 404 returned error can't find the container with id 63f09b2cb901d39afeea39f9a52088865a0d056b17e36a8e6a79fa1157b4e7ae Nov 03 03:46:28.393169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.393127 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:46:28.395241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.395209 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:28.395318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.395274 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:28.395318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.395302 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:46:28.917289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.917257 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerStarted","Data":"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d"} Nov 03 03:46:28.917721 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.917308 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerStarted","Data":"63f09b2cb901d39afeea39f9a52088865a0d056b17e36a8e6a79fa1157b4e7ae"} Nov 03 03:46:28.947951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:28.947913 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:29.041569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:29.041518 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podStartSLOduration=2.041503392 podStartE2EDuration="2.041503392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:29.037241795 +0000 UTC m=+2680.189930222" watchObservedRunningTime="2026-11-03 03:46:29.041503392 +0000 UTC m=+2680.194191813" Nov 03 03:46:29.955105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:29.955075 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:30.930774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:30.930745 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:30.931607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:30.931588 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:46:32.331444 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:32.331402 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:46:32.877674 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:32.877642 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:46:32.877881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:32.877713 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:46:47.329013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:47.328978 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:46:47.329525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:47.329201 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" containerID="cri-o://a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753" gracePeriod=30 Nov 03 03:46:49.210240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:49.210198 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:46:49.210741 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:49.210267 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:46:49.651234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:49.651203 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:46:49.661258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:49.661227 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:46:52.614417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.614390 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:46:52.753743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.753708 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.753991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.753828 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.753991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.753904 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.753991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.753964 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754092 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754277 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:52.754323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754268 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:52.754528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754353 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:52.754528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754399 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754467 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754543 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:52.754727 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754570 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754796 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\") pod \"192a40d2-037f-4757-919b-dc11688e9b19\" (UID: \"192a40d2-037f-4757-919b-dc11688e9b19\") " Nov 03 03:46:52.754879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.754808 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:52.755413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.755381 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.755413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.755411 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.755413 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.755427 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.755815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.755441 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192a40d2-037f-4757-919b-dc11688e9b19-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.755815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.755455 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.756800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.756762 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:46:52.756931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.756902 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "kube-api-access-nx8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:52.757278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.757257 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:52.757531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.757509 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token" pod "192a40d2-037f-4757-919b-dc11688e9b19" (UID: "192a40d2-037f-4757-919b-dc11688e9b19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:52.857062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.856967 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192a40d2-037f-4757-919b-dc11688e9b19-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.857062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.856999 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8d9\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-kube-api-access-nx8d9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.857062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.857011 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192a40d2-037f-4757-919b-dc11688e9b19-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:52.857062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:52.857019 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192a40d2-037f-4757-919b-dc11688e9b19-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:53.100495 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.100458 5275 generic.go:356] "Generic (PLEG): container finished" podID="192a40d2-037f-4757-919b-dc11688e9b19" containerID="a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753" exitCode=0 Nov 03 03:46:53.100687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.100527 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" Nov 03 03:46:53.100687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.100539 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerDied","Data":"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753"} Nov 03 03:46:53.100687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.100596 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449" event={"ID":"192a40d2-037f-4757-919b-dc11688e9b19","Type":"ContainerDied","Data":"4b9d53c4d6aa6b1ef31a5df0798713069b81e1ff8218c14963c05e0ee86d9b38"} Nov 03 03:46:53.100687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.100620 5275 scope.go:122] "RemoveContainer" containerID="a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753" Nov 03 03:46:53.111671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.111648 5275 scope.go:122] "RemoveContainer" containerID="834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d" Nov 03 03:46:53.121486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.121464 5275 scope.go:122] "RemoveContainer" containerID="a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753" Nov 03 03:46:53.121821 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:53.121801 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753\": container with ID starting with a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753 not found: ID does not exist" containerID="a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753" Nov 03 03:46:53.121904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.121830 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753"} err="failed to get container status \"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753\": rpc error: code = NotFound desc = could not find container \"a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753\": container with ID starting with a6dd475dc3b5cfa3d9fafb411fa0edb76f6aa73b7e27589d9b00ff23e0893753 not found: ID does not exist" Nov 03 03:46:53.121904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.121880 5275 scope.go:122] "RemoveContainer" containerID="834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d" Nov 03 03:46:53.122140 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:53.122121 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d\": container with ID starting with 834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d not found: ID does not exist" containerID="834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d" Nov 03 03:46:53.122178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.122147 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d"} err="failed to get container status \"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d\": rpc error: code = NotFound desc = could not find container \"834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d\": container with ID starting with 834133b1cb3f264f8e41b8c162feb015fbdf1fa38819ecda5481be4cc5714b2d not found: ID does not exist" Nov 03 03:46:53.268291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.268250 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:46:53.322300 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.322267 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b755ee-openshift-default-76c69986f-9b449"] Nov 03 03:46:53.421376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.421271 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:46:53.421636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:53.421608 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" containerID="cri-o://2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11" gracePeriod=30 Nov 03 03:46:54.985597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:54.985565 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192a40d2-037f-4757-919b-dc11688e9b19" path="/var/lib/kubelet/pods/192a40d2-037f-4757-919b-dc11688e9b19/volumes" Nov 03 03:46:55.744789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:55.744751 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-j4jzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:46:55.745002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:55.744816 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:46:58.699487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.699411 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:46:58.731176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731139 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731240 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731296 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731327 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731346 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731366 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731411 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731492 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731523 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\") pod \"c1833e32-0f17-4659-a75b-4c490cec5c7f\" (UID: \"c1833e32-0f17-4659-a75b-4c490cec5c7f\") " Nov 03 03:46:58.731742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731552 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:58.731742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731597 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:58.731742 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731711 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:58.731959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731730 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:58.732027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.731958 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:58.732177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.732127 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.732177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.732151 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1833e32-0f17-4659-a75b-4c490cec5c7f-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.732177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.732166 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.732177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.732180 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.732177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.732195 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.733804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.733781 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:58.733948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.733811 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:58.733948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.733929 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "kube-api-access-wq5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:58.733948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.733929 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo" pod "c1833e32-0f17-4659-a75b-4c490cec5c7f" (UID: "c1833e32-0f17-4659-a75b-4c490cec5c7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:46:58.833321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.833291 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.833321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.833321 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5tk\" (UniqueName: \"kubernetes.io/projected/c1833e32-0f17-4659-a75b-4c490cec5c7f-kube-api-access-wq5tk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.833536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.833331 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:58.833536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:58.833340 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1833e32-0f17-4659-a75b-4c490cec5c7f-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:59.141762 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.141728 5275 generic.go:356] "Generic (PLEG): container finished" podID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerID="2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11" exitCode=0 Nov 03 03:46:59.142005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.141777 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" Nov 03 03:46:59.142005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.141817 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" event={"ID":"c1833e32-0f17-4659-a75b-4c490cec5c7f","Type":"ContainerDied","Data":"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11"} Nov 03 03:46:59.142005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.141902 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb" event={"ID":"c1833e32-0f17-4659-a75b-4c490cec5c7f","Type":"ContainerDied","Data":"febe7aae4051bb6d0f6910da6fbb4f26a224fc6c20d4736253402087d481dce9"} Nov 03 03:46:59.142005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.141938 5275 scope.go:122] "RemoveContainer" containerID="2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11" Nov 03 03:46:59.152173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.152148 5275 scope.go:122] "RemoveContainer" containerID="2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11" Nov 03 03:46:59.152503 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:46:59.152483 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11\": container with ID starting with 2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11 not found: ID does not exist" containerID="2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11" Nov 03 03:46:59.152578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.152511 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11"} err="failed to get container status \"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11\": rpc error: code = NotFound desc = could not find container \"2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11\": container with ID starting with 2abde5edf4915c84eb5c0cbfa313f48bec6de87dd6653b8e3417ba4b585d3c11 not found: ID does not exist" Nov 03 03:46:59.306131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.306097 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:46:59.368873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:46:59.368410 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-j4jzb"] Nov 03 03:47:00.987987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:00.987954 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" path="/var/lib/kubelet/pods/c1833e32-0f17-4659-a75b-4c490cec5c7f/volumes" Nov 03 03:47:02.330640 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:47:02.330605 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:47:03.661956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:03.661918 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:47:03.662479 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:03.662196 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" containerID="cri-o://89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a" gracePeriod=30 Nov 03 03:47:07.956032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:07.955988 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:47:07.956406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:07.956062 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:47:08.916660 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:08.916633 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:47:09.060083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060042 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060109 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060157 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060191 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060256 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060290 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060341 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060380 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060407 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:47:09.060591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060460 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\") pod \"f48f0011-819d-4ce3-be7e-de2f91107715\" (UID: \"f48f0011-819d-4ce3-be7e-de2f91107715\") " Nov 03 03:47:09.061345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060704 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:47:09.061345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.060724 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:47:09.061345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061022 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:47:09.061606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061577 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.061735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061608 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.061735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061624 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.061735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061641 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48f0011-819d-4ce3-be7e-de2f91107715-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.061735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.061578 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:47:09.062614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.062583 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:47:09.062751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.062733 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:09.062925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.062909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "kube-api-access-qdbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:09.063825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.063810 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo" pod "f48f0011-819d-4ce3-be7e-de2f91107715" (UID: "f48f0011-819d-4ce3-be7e-de2f91107715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:47:09.163148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.163110 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48f0011-819d-4ce3-be7e-de2f91107715-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.163148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.163144 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.163148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.163153 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48f0011-819d-4ce3-be7e-de2f91107715-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.163405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.163162 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.163405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.163171 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbzh\" (UniqueName: \"kubernetes.io/projected/f48f0011-819d-4ce3-be7e-de2f91107715-kube-api-access-qdbzh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:09.217185 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.217149 5275 generic.go:356] "Generic (PLEG): container finished" podID="f48f0011-819d-4ce3-be7e-de2f91107715" containerID="89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a" exitCode=0 Nov 03 03:47:09.217398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.217192 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerDied","Data":"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a"} Nov 03 03:47:09.217398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.217199 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" Nov 03 03:47:09.217398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.217244 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5" event={"ID":"f48f0011-819d-4ce3-be7e-de2f91107715","Type":"ContainerDied","Data":"4eccb3ca94d250017ced2695eefdd223f4721e6a02c9c91887577ae0f655119f"} Nov 03 03:47:09.217398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.217271 5275 scope.go:122] "RemoveContainer" containerID="89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a" Nov 03 03:47:09.228444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.228422 5275 scope.go:122] "RemoveContainer" containerID="914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e" Nov 03 03:47:09.238889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.238864 5275 scope.go:122] "RemoveContainer" containerID="89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a" Nov 03 03:47:09.239175 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:47:09.239156 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a\": container with ID starting with 89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a not found: ID does not exist" containerID="89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a" Nov 03 03:47:09.239249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.239181 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a"} err="failed to get container status \"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a\": rpc error: code = NotFound desc = could not find container \"89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a\": container with ID starting with 89e4123ed54480300e319ee0c1cfdb25422e133f4f1494bddfe66fe1d2684c6a not found: ID does not exist" Nov 03 03:47:09.239249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.239198 5275 scope.go:122] "RemoveContainer" containerID="914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e" Nov 03 03:47:09.239437 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:47:09.239416 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e\": container with ID starting with 914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e not found: ID does not exist" containerID="914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e" Nov 03 03:47:09.239505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.239443 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e"} err="failed to get container status \"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e\": rpc error: code = NotFound desc = could not find container \"914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e\": container with ID starting with 914b3e9735e0b56e4f0b7751b8ade4a37c9a625b11e40a0b56a6d6ecfdd1246e not found: ID does not exist" Nov 03 03:47:09.383125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.383031 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:47:09.386079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:09.386050 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3480a2-openshift-default-7bd847fd85-k4jd5"] Nov 03 03:47:10.984336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:10.984300 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" path="/var/lib/kubelet/pods/f48f0011-819d-4ce3-be7e-de2f91107715/volumes" Nov 03 03:47:17.872442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.872399 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2"] Nov 03 03:47:17.873245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873225 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:17.873307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873249 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:17.873307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873272 5275 cpu_manager.go:414] "containerMap: removing container" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.873307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873280 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.873307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873293 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" Nov 03 03:47:17.873307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873301 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" Nov 03 03:47:17.873470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873312 5275 cpu_manager.go:414] "containerMap: removing container" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.873470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873318 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.873470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873377 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:17.873470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873385 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:17.873594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873499 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:17.873594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873515 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.873594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873528 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1833e32-0f17-4659-a75b-4c490cec5c7f" containerName="istio-proxy" Nov 03 03:47:17.873594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.873538 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="192a40d2-037f-4757-919b-dc11688e9b19" containerName="istio-proxy" Nov 03 03:47:17.878974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.878940 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:17.932713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.932684 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2"] Nov 03 03:47:17.978733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.978693 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:47:17.978733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.978712 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:47:17.980599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.980574 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:47:17.980599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:17.980580 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:47:18.063607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.063564 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxltj\" (UniqueName: \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.063833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.063699 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.063833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.063789 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.165450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.165353 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.165450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.165427 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxltj\" (UniqueName: \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.165684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.165500 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.166084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.166059 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.168141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.168111 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.226985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.226957 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxltj\" (UniqueName: \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\") pod \"maas-api-revoke-sub-6766f24a-p4tb2\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.493724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.493632 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:18.873750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:18.873647 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2"] Nov 03 03:47:18.877017 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:47:18.876971 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cf7639_c16e_4add_8737_58ec1b14238b.slice/crio-6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226 WatchSource:0}: Error finding container 6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226: Status 404 returned error can't find the container with id 6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226 Nov 03 03:47:19.302827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:19.302791 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" event={"ID":"91cf7639-c16e-4add-8737-58ec1b14238b","Type":"ContainerStarted","Data":"b6b5045ad35a3fbad638711b48efd62ed61aebd84f52a71fe404a95f052ff937"} Nov 03 03:47:19.302996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:19.302842 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" event={"ID":"91cf7639-c16e-4add-8737-58ec1b14238b","Type":"ContainerStarted","Data":"6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226"} Nov 03 03:47:19.419821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:19.419685 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" podStartSLOduration=2.419668185 podStartE2EDuration="2.419668185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:47:19.418809847 +0000 UTC m=+2730.571498292" watchObservedRunningTime="2026-11-03 03:47:19.419668185 +0000 UTC m=+2730.572356607" Nov 03 03:47:20.312201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:20.312164 5275 generic.go:356] "Generic (PLEG): container finished" podID="91cf7639-c16e-4add-8737-58ec1b14238b" containerID="b6b5045ad35a3fbad638711b48efd62ed61aebd84f52a71fe404a95f052ff937" exitCode=0 Nov 03 03:47:20.312662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:20.312247 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" event={"ID":"91cf7639-c16e-4add-8737-58ec1b14238b","Type":"ContainerDied","Data":"b6b5045ad35a3fbad638711b48efd62ed61aebd84f52a71fe404a95f052ff937"} Nov 03 03:47:21.464017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.463991 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:21.612482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.612392 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\") pod \"91cf7639-c16e-4add-8737-58ec1b14238b\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " Nov 03 03:47:21.612482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.612450 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\") pod \"91cf7639-c16e-4add-8737-58ec1b14238b\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " Nov 03 03:47:21.612683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.612518 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\" (UniqueName: \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\") pod \"91cf7639-c16e-4add-8737-58ec1b14238b\" (UID: \"91cf7639-c16e-4add-8737-58ec1b14238b\") " Nov 03 03:47:21.612994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.612964 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca" pod "91cf7639-c16e-4add-8737-58ec1b14238b" (UID: "91cf7639-c16e-4add-8737-58ec1b14238b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:47:21.614839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.614811 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca" pod "91cf7639-c16e-4add-8737-58ec1b14238b" (UID: "91cf7639-c16e-4add-8737-58ec1b14238b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:47:21.614967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.614884 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj" pod "91cf7639-c16e-4add-8737-58ec1b14238b" (UID: "91cf7639-c16e-4add-8737-58ec1b14238b"). InnerVolumeSpecName "kube-api-access-sxltj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:21.713718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.713678 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxltj\" (UniqueName: \"kubernetes.io/projected/91cf7639-c16e-4add-8737-58ec1b14238b-kube-api-access-sxltj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:21.713718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.713714 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:21.713718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:21.713724 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cf7639-c16e-4add-8737-58ec1b14238b-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:22.327153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:22.327115 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" event={"ID":"91cf7639-c16e-4add-8737-58ec1b14238b","Type":"ContainerDied","Data":"6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226"} Nov 03 03:47:22.327333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:22.327167 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3584beb070205661be6e504e4faeef3964111e04f54a7cad87b2b18442a226" Nov 03 03:47:22.327333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:22.327130 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2" Nov 03 03:47:32.329929 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:47:32.329892 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:47:36.244340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.244298 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt"] Nov 03 03:47:36.245064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.245015 5275 cpu_manager.go:414] "containerMap: removing container" podUID="91cf7639-c16e-4add-8737-58ec1b14238b" containerName="revoke-keys" Nov 03 03:47:36.245166 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.245063 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cf7639-c16e-4add-8737-58ec1b14238b" containerName="revoke-keys" Nov 03 03:47:36.245231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.245223 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48f0011-819d-4ce3-be7e-de2f91107715" containerName="istio-proxy" Nov 03 03:47:36.245286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.245235 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cf7639-c16e-4add-8737-58ec1b14238b" containerName="revoke-keys" Nov 03 03:47:36.248863 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.248828 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.302385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.302349 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:47:36.352050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.352019 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:47:36.352289 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.352040 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:47:36.352393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.352101 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:47:36.355064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.355036 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt"] Nov 03 03:47:36.379039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.379000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.379237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.379124 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzzw\" (UniqueName: \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.379237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.379214 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.480463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.480417 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.480689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.480494 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzzw\" (UniqueName: \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.480689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.480570 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.481312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.481282 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.483337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.483308 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.541758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.541727 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzzw\" (UniqueName: \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\") pod \"maas-api-revoke-sub-f02f2f45-kbjmt\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.565055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.565022 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:36.947893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:36.947860 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt"] Nov 03 03:47:36.949793 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:47:36.949764 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323a16bc_ba7c_4cd4_abee_d987bcb79a05.slice/crio-5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2 WatchSource:0}: Error finding container 5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2: Status 404 returned error can't find the container with id 5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2 Nov 03 03:47:37.432646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:37.432606 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" event={"ID":"323a16bc-ba7c-4cd4-abee-d987bcb79a05","Type":"ContainerStarted","Data":"ac253e1c32f254a19e68724e3d109e4e665b9cc052a6d516069e66528a5f6275"} Nov 03 03:47:37.433107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:37.432658 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" event={"ID":"323a16bc-ba7c-4cd4-abee-d987bcb79a05","Type":"ContainerStarted","Data":"5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2"} Nov 03 03:47:37.550668 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:37.550593 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" podStartSLOduration=1.550571498 podStartE2EDuration="1.550571498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:47:37.547116578 +0000 UTC m=+2748.699804999" watchObservedRunningTime="2026-11-03 03:47:37.550571498 +0000 UTC m=+2748.703259922" Nov 03 03:47:38.443279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:38.443243 5275 generic.go:356] "Generic (PLEG): container finished" podID="323a16bc-ba7c-4cd4-abee-d987bcb79a05" containerID="ac253e1c32f254a19e68724e3d109e4e665b9cc052a6d516069e66528a5f6275" exitCode=0 Nov 03 03:47:38.443710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:38.443324 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" event={"ID":"323a16bc-ba7c-4cd4-abee-d987bcb79a05","Type":"ContainerDied","Data":"ac253e1c32f254a19e68724e3d109e4e665b9cc052a6d516069e66528a5f6275"} Nov 03 03:47:39.639694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.639664 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:39.722391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.722352 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\") pod \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " Nov 03 03:47:39.722611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.722445 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\") pod \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " Nov 03 03:47:39.722611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.722574 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\" (UniqueName: \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\") pod \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\" (UID: \"323a16bc-ba7c-4cd4-abee-d987bcb79a05\") " Nov 03 03:47:39.722901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.722821 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca" pod "323a16bc-ba7c-4cd4-abee-d987bcb79a05" (UID: "323a16bc-ba7c-4cd4-abee-d987bcb79a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:47:39.723086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.723063 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:39.724777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.724749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca" pod "323a16bc-ba7c-4cd4-abee-d987bcb79a05" (UID: "323a16bc-ba7c-4cd4-abee-d987bcb79a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:47:39.724917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.724803 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw" pod "323a16bc-ba7c-4cd4-abee-d987bcb79a05" (UID: "323a16bc-ba7c-4cd4-abee-d987bcb79a05"). InnerVolumeSpecName "kube-api-access-btzzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:39.824045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.824006 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323a16bc-ba7c-4cd4-abee-d987bcb79a05-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:39.824045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:39.824039 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btzzw\" (UniqueName: \"kubernetes.io/projected/323a16bc-ba7c-4cd4-abee-d987bcb79a05-kube-api-access-btzzw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:40.489645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:40.489610 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" Nov 03 03:47:40.489883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:40.489636 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt" event={"ID":"323a16bc-ba7c-4cd4-abee-d987bcb79a05","Type":"ContainerDied","Data":"5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2"} Nov 03 03:47:40.489883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:40.489723 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5453f0c01965adfdd1c6bf82a5fa045b15de53fbb0127af5cbeec7be6141ccd2" Nov 03 03:47:50.576097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.576065 5275 generic.go:356] "Generic (PLEG): container finished" podID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerID="5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b" exitCode=0 Nov 03 03:47:50.576538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.576130 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerDied","Data":"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b"} Nov 03 03:47:50.576538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.576495 5275 scope.go:122] "RemoveContainer" containerID="5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b" Nov 03 03:47:50.577904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.577869 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:47:50.577984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.577944 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:47:50.577984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:50.577976 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:47:51.584659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:51.584624 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerStarted","Data":"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba"} Nov 03 03:47:52.012097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.012018 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:47:52.012260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.012229 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 03:47:52.012320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.012298 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 03:47:52.678038 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.677999 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l"] Nov 03 03:47:52.678748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.678725 5275 cpu_manager.go:414] "containerMap: removing container" podUID="323a16bc-ba7c-4cd4-abee-d987bcb79a05" containerName="revoke-keys" Nov 03 03:47:52.678886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.678749 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323a16bc-ba7c-4cd4-abee-d987bcb79a05" containerName="revoke-keys" Nov 03 03:47:52.678973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.678956 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="323a16bc-ba7c-4cd4-abee-d987bcb79a05" containerName="revoke-keys" Nov 03 03:47:52.683665 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.683643 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.721604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.721571 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l"] Nov 03 03:47:52.736468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.736435 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.736663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.736496 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.736739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.736686 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxbn\" (UniqueName: \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.775151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.775115 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:47:52.775367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.775226 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:47:52.775367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.775324 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:47:52.775367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.775115 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:47:52.837907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.837869 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxbn\" (UniqueName: \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.838119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.837987 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.838119 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.838065 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.838793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.838770 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.840681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.840656 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.906408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.906372 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxbn\" (UniqueName: \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\") pod \"maas-api-revoke-sub-a245853b-c9t5l\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:52.998029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:52.997945 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:53.012454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:53.012423 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 03:47:53.012549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:53.012503 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 03:47:53.413764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:53.413720 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l"] Nov 03 03:47:53.417864 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:47:53.415197 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921df2ef_0608_4a9f_a43f_cfebeeaff6c9.slice/crio-206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458 WatchSource:0}: Error finding container 206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458: Status 404 returned error can't find the container with id 206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458 Nov 03 03:47:53.598939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:53.598903 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" event={"ID":"921df2ef-0608-4a9f-a43f-cfebeeaff6c9","Type":"ContainerStarted","Data":"206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458"} Nov 03 03:47:54.011947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:54.011833 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 03:47:54.012340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:54.011939 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 03:47:54.608350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:54.608302 5275 generic.go:356] "Generic (PLEG): container finished" podID="921df2ef-0608-4a9f-a43f-cfebeeaff6c9" containerID="40af18ca561faab2c98350b8ff1165e6abc342ed433e8e545dcde04c25ed3c49" exitCode=0 Nov 03 03:47:54.608545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:54.608410 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" event={"ID":"921df2ef-0608-4a9f-a43f-cfebeeaff6c9","Type":"ContainerDied","Data":"40af18ca561faab2c98350b8ff1165e6abc342ed433e8e545dcde04c25ed3c49"} Nov 03 03:47:55.012391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.012304 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 03:47:55.012779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.012385 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 03:47:55.795968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.795939 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:55.890382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.890291 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\") pod \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " Nov 03 03:47:55.890382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.890360 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\" (UniqueName: \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\") pod \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " Nov 03 03:47:55.890591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.890436 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\") pod \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\" (UID: \"921df2ef-0608-4a9f-a43f-cfebeeaff6c9\") " Nov 03 03:47:55.890773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.890749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca" pod "921df2ef-0608-4a9f-a43f-cfebeeaff6c9" (UID: "921df2ef-0608-4a9f-a43f-cfebeeaff6c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:47:55.892654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.892629 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca" pod "921df2ef-0608-4a9f-a43f-cfebeeaff6c9" (UID: "921df2ef-0608-4a9f-a43f-cfebeeaff6c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:47:55.892959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.892939 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn" pod "921df2ef-0608-4a9f-a43f-cfebeeaff6c9" (UID: "921df2ef-0608-4a9f-a43f-cfebeeaff6c9"). InnerVolumeSpecName "kube-api-access-4jxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:55.992124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.992092 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:55.992124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.992121 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:55.992124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:55.992131 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxbn\" (UniqueName: \"kubernetes.io/projected/921df2ef-0608-4a9f-a43f-cfebeeaff6c9-kube-api-access-4jxbn\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:56.012096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:56.012060 5275 patch_prober.go:29] interesting pod/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 03:47:56.012244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:56.012131 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 03:47:56.623222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:56.623191 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" Nov 03 03:47:56.623654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:56.623218 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l" event={"ID":"921df2ef-0608-4a9f-a43f-cfebeeaff6c9","Type":"ContainerDied","Data":"206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458"} Nov 03 03:47:56.623654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:56.623266 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="206673b4321df773009c6d513457240d9d7f97a5fca7b44db3b017e7fe40a458" Nov 03 03:47:57.016365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:57.016270 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:47:57.016702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:57.016676 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:47:57.017542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:47:57.017520 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:48:02.331487 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:48:02.331449 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:48:02.839724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:02.839689 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:48:02.839985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:02.839960 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" containerID="cri-o://d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9" gracePeriod=30 Nov 03 03:48:06.003900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:06.003830 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:48:06.004399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:06.003938 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:48:08.108730 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.108654 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150433 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150533 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150583 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150636 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150731 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150780 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150817 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.150985 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.151020 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\") pod \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\" (UID: \"93f9c7aa-fff2-4a4e-acff-0da6c71209c0\") " Nov 03 03:48:08.154776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.154436 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:48:08.154776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.154618 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:48:08.154935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.154775 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:48:08.155624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.155265 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:48:08.155624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.155585 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:08.159705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.159670 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:48:08.159963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.159921 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:48:08.160084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.160056 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:08.160271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.160237 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z" pod "93f9c7aa-fff2-4a4e-acff-0da6c71209c0" (UID: "93f9c7aa-fff2-4a4e-acff-0da6c71209c0"). InnerVolumeSpecName "kube-api-access-4549z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:08.253263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253228 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253258 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253268 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4549z\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-kube-api-access-4549z\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253278 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253287 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253295 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253303 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253311 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.253320 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f9c7aa-fff2-4a4e-acff-0da6c71209c0-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:08.716228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.716195 5275 generic.go:356] "Generic (PLEG): container finished" podID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerID="d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9" exitCode=0 Nov 03 03:48:08.716440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.716245 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" Nov 03 03:48:08.716440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.716279 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerDied","Data":"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9"} Nov 03 03:48:08.716440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.716342 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt" event={"ID":"93f9c7aa-fff2-4a4e-acff-0da6c71209c0","Type":"ContainerDied","Data":"e6d00ace46499c7de123c7342997da94d238427714de71128e9b8bdf6b497d55"} Nov 03 03:48:08.716440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.716371 5275 scope.go:122] "RemoveContainer" containerID="d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9" Nov 03 03:48:08.730243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.730216 5275 scope.go:122] "RemoveContainer" containerID="18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a" Nov 03 03:48:08.740576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.740557 5275 scope.go:122] "RemoveContainer" containerID="d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9" Nov 03 03:48:08.740893 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:48:08.740868 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9\": container with ID starting with d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9 not found: ID does not exist" containerID="d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9" Nov 03 03:48:08.740969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.740901 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9"} err="failed to get container status \"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9\": rpc error: code = NotFound desc = could not find container \"d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9\": container with ID starting with d0a2fd1a1cdd727a701a4b0bd7696027ff3a0a0f7cbb43da3298a41b1c66bfb9 not found: ID does not exist" Nov 03 03:48:08.740969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.740922 5275 scope.go:122] "RemoveContainer" containerID="18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a" Nov 03 03:48:08.741224 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:48:08.741203 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a\": container with ID starting with 18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a not found: ID does not exist" containerID="18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a" Nov 03 03:48:08.741271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.741230 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a"} err="failed to get container status \"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a\": rpc error: code = NotFound desc = could not find container \"18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a\": container with ID starting with 18f571ab68bbff87dcae1b108b575544e411fdabe20bc421e8dcc2171dbd3e4a not found: ID does not exist" Nov 03 03:48:08.782282 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.782247 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4"] Nov 03 03:48:08.783080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783060 5275 cpu_manager.go:414] "containerMap: removing container" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:08.783149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783083 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:08.783149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783123 5275 cpu_manager.go:414] "containerMap: removing container" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:08.783149 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783131 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:08.783255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783173 5275 cpu_manager.go:414] "containerMap: removing container" podUID="921df2ef-0608-4a9f-a43f-cfebeeaff6c9" containerName="revoke-keys" Nov 03 03:48:08.783255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921df2ef-0608-4a9f-a43f-cfebeeaff6c9" containerName="revoke-keys" Nov 03 03:48:08.783316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783299 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="921df2ef-0608-4a9f-a43f-cfebeeaff6c9" containerName="revoke-keys" Nov 03 03:48:08.783349 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.783317 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:08.788703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.788680 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.841097 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.841058 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:48:08.841288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.841109 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:48:08.841537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.841490 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:48:08.841643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.841501 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:48:08.842330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.842305 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4"] Nov 03 03:48:08.863895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.861926 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7qk\" (UniqueName: \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.863895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.861994 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.863895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.862031 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.887422 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.887387 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:48:08.891397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.891367 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a9e5cd-openshift-default-5b8c7c44d5-rfnzt"] Nov 03 03:48:08.963142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.963096 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7qk\" (UniqueName: \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.963355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.963150 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.963355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.963178 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.964030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.963999 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.965961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.965936 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:08.986989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:08.986905 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" path="/var/lib/kubelet/pods/93f9c7aa-fff2-4a4e-acff-0da6c71209c0/volumes" Nov 03 03:48:09.024906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:09.024835 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7qk\" (UniqueName: \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\") pod \"maas-api-revoke-sub-9382d83e-vmkp4\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:09.103572 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:09.103537 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:09.492210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:09.492069 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4"] Nov 03 03:48:09.492210 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:09.492155 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02322c9b_d923_467c_adbc_e65fc2511cf0.slice/crio-e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555 WatchSource:0}: Error finding container e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555: Status 404 returned error can't find the container with id e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555 Nov 03 03:48:09.726277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:09.726244 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" event={"ID":"02322c9b-d923-467c-adbc-e65fc2511cf0","Type":"ContainerStarted","Data":"e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555"} Nov 03 03:48:10.735708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:10.735611 5275 generic.go:356] "Generic (PLEG): container finished" podID="02322c9b-d923-467c-adbc-e65fc2511cf0" containerID="353bcd2e7a55911a5f8cf801fd42e3ccd598aca9d92d712ec8bcda6ad422f884" exitCode=0 Nov 03 03:48:10.735708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:10.735686 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" event={"ID":"02322c9b-d923-467c-adbc-e65fc2511cf0","Type":"ContainerDied","Data":"353bcd2e7a55911a5f8cf801fd42e3ccd598aca9d92d712ec8bcda6ad422f884"} Nov 03 03:48:11.991202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:11.991175 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:12.112872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.112672 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\") pod \"02322c9b-d923-467c-adbc-e65fc2511cf0\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " Nov 03 03:48:12.112872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.112812 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\" (UniqueName: \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\") pod \"02322c9b-d923-467c-adbc-e65fc2511cf0\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " Nov 03 03:48:12.113148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.112928 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\") pod \"02322c9b-d923-467c-adbc-e65fc2511cf0\" (UID: \"02322c9b-d923-467c-adbc-e65fc2511cf0\") " Nov 03 03:48:12.124873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.116694 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca" pod "02322c9b-d923-467c-adbc-e65fc2511cf0" (UID: "02322c9b-d923-467c-adbc-e65fc2511cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:12.124873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.119319 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca" pod "02322c9b-d923-467c-adbc-e65fc2511cf0" (UID: "02322c9b-d923-467c-adbc-e65fc2511cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:12.128077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.128026 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk" pod "02322c9b-d923-467c-adbc-e65fc2511cf0" (UID: "02322c9b-d923-467c-adbc-e65fc2511cf0"). InnerVolumeSpecName "kube-api-access-pq7qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:12.215096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.215038 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7qk\" (UniqueName: \"kubernetes.io/projected/02322c9b-d923-467c-adbc-e65fc2511cf0-kube-api-access-pq7qk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:12.215096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.215064 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:12.215096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.215075 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02322c9b-d923-467c-adbc-e65fc2511cf0-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:12.764876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.755045 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" event={"ID":"02322c9b-d923-467c-adbc-e65fc2511cf0","Type":"ContainerDied","Data":"e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555"} Nov 03 03:48:12.764876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.755113 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9678e72352d302ab23379e46800d28451f8350015101f4f0a7ffd0ccf9ff555" Nov 03 03:48:12.764876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:12.755192 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4" Nov 03 03:48:16.283899 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.283005 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:48:16.284424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.284064 5275 cpu_manager.go:414] "containerMap: removing container" podUID="02322c9b-d923-467c-adbc-e65fc2511cf0" containerName="revoke-keys" Nov 03 03:48:16.284424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.284098 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02322c9b-d923-467c-adbc-e65fc2511cf0" containerName="revoke-keys" Nov 03 03:48:16.284424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.284279 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f9c7aa-fff2-4a4e-acff-0da6c71209c0" containerName="istio-proxy" Nov 03 03:48:16.284424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.284296 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="02322c9b-d923-467c-adbc-e65fc2511cf0" containerName="revoke-keys" Nov 03 03:48:16.292295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.292255 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.357921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.344799 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8950af-openshift-default-dockercfg-vhcvh\"" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409343 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409391 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409443 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409514 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409557 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzh5q\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409602 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409635 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409690 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.416903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.409727 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.420238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.418400 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:48:16.510799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.510765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.510799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.510802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.510836 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.510915 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.510957 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzh5q\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511000 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511035 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511082 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511118 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511254 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511564 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.511983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.511831 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.512179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.512155 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.512365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.512347 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.513945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.513922 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.514335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.514317 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.541210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.541126 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs"] Nov 03 03:48:16.545293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.545268 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.580434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.578830 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzh5q\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.601048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.598729 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:48:16.601048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.598910 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:48:16.601048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.599002 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\") pod \"e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.601048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.599502 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:48:16.601048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.599601 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:48:16.603827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.603802 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs"] Nov 03 03:48:16.611949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.610414 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:16.714500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.714473 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.714664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.714511 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.714664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.714543 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wrz\" (UniqueName: \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.815832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.815747 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.815832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.815790 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.815832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.815820 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wrz\" (UniqueName: \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.816463 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.816442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.818380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.818359 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.881952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.879586 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wrz\" (UniqueName: \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\") pod \"maas-api-revoke-sub-d7038186-xrcjs\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:16.898285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:16.898248 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:17.090177 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:17.090148 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac72609c_50df_45e2_9b87_d5e426c55446.slice/crio-5b396acfc9f9527ecd3730a59694d6a4d79881c7c20486bde68dccc715494ab2 WatchSource:0}: Error finding container 5b396acfc9f9527ecd3730a59694d6a4d79881c7c20486bde68dccc715494ab2: Status 404 returned error can't find the container with id 5b396acfc9f9527ecd3730a59694d6a4d79881c7c20486bde68dccc715494ab2 Nov 03 03:48:17.091241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.091203 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:48:17.093135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.093101 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:17.093225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.093163 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:17.093225 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.093193 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:17.321728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.321691 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs"] Nov 03 03:48:17.322209 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:17.322184 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6239376e_e219_455f_adc6_f371e71bc6c1.slice/crio-fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806 WatchSource:0}: Error finding container fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806: Status 404 returned error can't find the container with id fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806 Nov 03 03:48:17.807159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.807122 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" event={"ID":"6239376e-e219-455f-adc6-f371e71bc6c1","Type":"ContainerStarted","Data":"5515fa668af79259d62637b7dafd7ade5c7e6cda2cf60de4ea6f1bc9ba4e07f4"} Nov 03 03:48:17.807352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.807190 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" event={"ID":"6239376e-e219-455f-adc6-f371e71bc6c1","Type":"ContainerStarted","Data":"fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806"} Nov 03 03:48:17.808933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.808908 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerStarted","Data":"d17fec1a99dfddab749d602d7f4a88d22b175135877b41e2f09fc8a111193864"} Nov 03 03:48:17.809096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.808954 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerStarted","Data":"5b396acfc9f9527ecd3730a59694d6a4d79881c7c20486bde68dccc715494ab2"} Nov 03 03:48:17.925618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:17.925513 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" podStartSLOduration=1.9254991970000002 podStartE2EDuration="1.925499197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:17.922097293 +0000 UTC m=+2789.074785714" watchObservedRunningTime="2026-11-03 03:48:17.925499197 +0000 UTC m=+2789.078187617" Nov 03 03:48:18.045509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.045455 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podStartSLOduration=2.045439681 podStartE2EDuration="2.045439681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:18.043085098 +0000 UTC m=+2789.195773519" watchObservedRunningTime="2026-11-03 03:48:18.045439681 +0000 UTC m=+2789.198128099" Nov 03 03:48:18.612600 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.612562 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:18.618694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.618664 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:18.820749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.820714 5275 generic.go:356] "Generic (PLEG): container finished" podID="6239376e-e219-455f-adc6-f371e71bc6c1" containerID="5515fa668af79259d62637b7dafd7ade5c7e6cda2cf60de4ea6f1bc9ba4e07f4" exitCode=0 Nov 03 03:48:18.820968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.820811 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" event={"ID":"6239376e-e219-455f-adc6-f371e71bc6c1","Type":"ContainerDied","Data":"5515fa668af79259d62637b7dafd7ade5c7e6cda2cf60de4ea6f1bc9ba4e07f4"} Nov 03 03:48:18.821251 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.821227 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:18.822202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:18.822181 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:48:19.992555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:19.992512 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:20.159421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.159152 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\") pod \"6239376e-e219-455f-adc6-f371e71bc6c1\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " Nov 03 03:48:20.159421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.159410 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\") pod \"6239376e-e219-455f-adc6-f371e71bc6c1\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " Nov 03 03:48:20.159655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.159462 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\" (UniqueName: \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\") pod \"6239376e-e219-455f-adc6-f371e71bc6c1\" (UID: \"6239376e-e219-455f-adc6-f371e71bc6c1\") " Nov 03 03:48:20.159655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.159614 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca" pod "6239376e-e219-455f-adc6-f371e71bc6c1" (UID: "6239376e-e219-455f-adc6-f371e71bc6c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:20.160501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.160471 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:20.162445 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.162414 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz" pod "6239376e-e219-455f-adc6-f371e71bc6c1" (UID: "6239376e-e219-455f-adc6-f371e71bc6c1"). InnerVolumeSpecName "kube-api-access-k6wrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:20.162555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.162507 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca" pod "6239376e-e219-455f-adc6-f371e71bc6c1" (UID: "6239376e-e219-455f-adc6-f371e71bc6c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:20.261640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.261596 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6wrz\" (UniqueName: \"kubernetes.io/projected/6239376e-e219-455f-adc6-f371e71bc6c1-kube-api-access-k6wrz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:20.261640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.261638 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6239376e-e219-455f-adc6-f371e71bc6c1-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:20.840040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.840007 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" Nov 03 03:48:20.842223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.840033 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs" event={"ID":"6239376e-e219-455f-adc6-f371e71bc6c1","Type":"ContainerDied","Data":"fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806"} Nov 03 03:48:20.842223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:20.840091 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4a7de2a85837da4b78169bb33d47022d12c632b20586b86ab5b1f79b073806" Nov 03 03:48:21.848170 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.848140 5275 generic.go:356] "Generic (PLEG): container finished" podID="7c821073-1948-4520-98e5-ca080b896b1a" containerID="e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d" exitCode=0 Nov 03 03:48:21.848581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.848210 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerDied","Data":"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d"} Nov 03 03:48:21.848637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.848619 5275 scope.go:122] "RemoveContainer" containerID="e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d" Nov 03 03:48:21.850108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.850082 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:21.850177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.850152 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:21.850214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:21.850185 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:48:22.879994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:22.879912 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerStarted","Data":"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662"} Nov 03 03:48:22.947907 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:22.947813 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:48:22.948092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:22.948039 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:22.948146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:22.948092 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:23.948456 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:23.948422 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:23.948952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:23.948481 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:24.574788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.574751 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:48:24.575608 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.575581 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6239376e-e219-455f-adc6-f371e71bc6c1" containerName="revoke-keys" Nov 03 03:48:24.575792 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.575763 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6239376e-e219-455f-adc6-f371e71bc6c1" containerName="revoke-keys" Nov 03 03:48:24.576290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.576128 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6239376e-e219-455f-adc6-f371e71bc6c1" containerName="revoke-keys" Nov 03 03:48:24.580030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.580008 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.631272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.631221 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:48:24.676679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.676648 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8950af-dockercfg-fvhwc\"" Nov 03 03:48:24.677063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.677041 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8950af\"" Nov 03 03:48:24.704159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.704125 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:48:24.709585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.709557 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.715857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.715762 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw5p\" (UniqueName: \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.716035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.715988 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.716133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.716039 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9kb\" (UniqueName: \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.716133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.716123 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.777596 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.777558 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:48:24.817527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.817485 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.817724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.817559 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw5p\" (UniqueName: \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.817724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.817691 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.817885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.817758 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9kb\" (UniqueName: \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.818904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.818874 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.819015 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.818928 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.880035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.879918 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9kb\" (UniqueName: \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\") pod \"payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:24.880035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.879939 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw5p\" (UniqueName: \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\") pod \"payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.894010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.893974 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:24.948354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.948317 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:24.948533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:24.948379 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:25.029424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.029386 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:25.236026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.235942 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv"] Nov 03 03:48:25.241939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.241915 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.295901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.294669 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:48:25.295901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.294695 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:48:25.295901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.294747 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:48:25.295901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.294752 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:48:25.297483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.297453 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv"] Nov 03 03:48:25.324259 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.324229 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.324416 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.324392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbt47\" (UniqueName: \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.324492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.324419 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.348432 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.348404 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:48:25.348550 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:25.348434 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e91555_f47b_4a27_9d64_dba2ec8637df.slice/crio-fe53df76c1df6e7ca1240c51fe63a6822df20c1b7a2cd82b3d9a56e9a08d6462 WatchSource:0}: Error finding container fe53df76c1df6e7ca1240c51fe63a6822df20c1b7a2cd82b3d9a56e9a08d6462: Status 404 returned error can't find the container with id fe53df76c1df6e7ca1240c51fe63a6822df20c1b7a2cd82b3d9a56e9a08d6462 Nov 03 03:48:25.425142 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.425112 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.425319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.425260 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbt47\" (UniqueName: \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.425319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.425289 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.426105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.426080 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.428158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.428133 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.470012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.468810 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:48:25.487841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.487816 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbt47\" (UniqueName: \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\") pod \"maas-api-revoke-sub-8ea4521e-82phv\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.583388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.583360 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:25.914540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.914489 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" event={"ID":"c6e91555-f47b-4a27-9d64-dba2ec8637df","Type":"ContainerStarted","Data":"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e"} Nov 03 03:48:25.914752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.914614 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" event={"ID":"c6e91555-f47b-4a27-9d64-dba2ec8637df","Type":"ContainerStarted","Data":"fe53df76c1df6e7ca1240c51fe63a6822df20c1b7a2cd82b3d9a56e9a08d6462"} Nov 03 03:48:25.916691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.916664 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" event={"ID":"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba","Type":"ContainerStarted","Data":"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5"} Nov 03 03:48:25.916912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.916895 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" event={"ID":"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba","Type":"ContainerStarted","Data":"408423cf50a491b3a0916123de792c25cddc9dbc905d7fe57543666ef0542885"} Nov 03 03:48:25.917025 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.917015 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:25.948574 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.948535 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:25.949013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.948588 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:25.985046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:25.981627 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv"] Nov 03 03:48:25.985046 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:25.983435 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7988fb_b018_4c68_a9aa_f392e92ff16c.slice/crio-14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123 WatchSource:0}: Error finding container 14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123: Status 404 returned error can't find the container with id 14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123 Nov 03 03:48:26.080661 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.080600 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" podStartSLOduration=2.080580913 podStartE2EDuration="2.080580913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:26.045800198 +0000 UTC m=+2797.198488619" watchObservedRunningTime="2026-11-03 03:48:26.080580913 +0000 UTC m=+2797.233269376" Nov 03 03:48:26.926013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.925976 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" event={"ID":"6a7988fb-b018-4c68-a9aa-f392e92ff16c","Type":"ContainerStarted","Data":"f32e2f11a68d416dc868b8004ad6658267a05c859b1aecc062d312581d050379"} Nov 03 03:48:26.926199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.926129 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" event={"ID":"6a7988fb-b018-4c68-a9aa-f392e92ff16c","Type":"ContainerStarted","Data":"14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123"} Nov 03 03:48:26.926199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.926163 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:26.948734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.948701 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:26.949206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:26.948758 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:27.044417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.044357 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" podStartSLOduration=3.044337768 podStartE2EDuration="3.044337768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:26.169538627 +0000 UTC m=+2797.322227048" watchObservedRunningTime="2026-11-03 03:48:27.044337768 +0000 UTC m=+2798.197026199" Nov 03 03:48:27.044879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.044814 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" podStartSLOduration=2.044801569 podStartE2EDuration="2.044801569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:27.043987364 +0000 UTC m=+2798.196675782" watchObservedRunningTime="2026-11-03 03:48:27.044801569 +0000 UTC m=+2798.197490026" Nov 03 03:48:27.477873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.474747 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms"] Nov 03 03:48:27.482355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.482329 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.513162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.513125 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms"] Nov 03 03:48:27.573876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.573598 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6gl\" (UniqueName: \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.573876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.573744 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.573876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.573778 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.674865 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.674814 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6gl\" (UniqueName: \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.675049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.674983 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.675049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.675015 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.675626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.675605 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.678017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.677991 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.735834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.735743 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6gl\" (UniqueName: \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\") pod \"maas-api-revoke-sub-2591e416-qwhms\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.797283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.797247 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:27.933585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.933548 5275 generic.go:356] "Generic (PLEG): container finished" podID="6a7988fb-b018-4c68-a9aa-f392e92ff16c" containerID="f32e2f11a68d416dc868b8004ad6658267a05c859b1aecc062d312581d050379" exitCode=0 Nov 03 03:48:27.933735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.933633 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" event={"ID":"6a7988fb-b018-4c68-a9aa-f392e92ff16c","Type":"ContainerDied","Data":"f32e2f11a68d416dc868b8004ad6658267a05c859b1aecc062d312581d050379"} Nov 03 03:48:27.948087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.948049 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 03:48:27.948297 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:27.948116 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 03:48:28.178671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.178640 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms"] Nov 03 03:48:28.179608 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:28.179576 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b2bb35_55e9_4380_a73b_fd3f4ce541b3.slice/crio-d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a WatchSource:0}: Error finding container d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a: Status 404 returned error can't find the container with id d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a Nov 03 03:48:28.947427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.947391 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" event={"ID":"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3","Type":"ContainerStarted","Data":"c80f8cff525cb02d529c8800111e15eb6c30ce9677d6b5322936004e1e1a9bf1"} Nov 03 03:48:28.947621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.947452 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" event={"ID":"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3","Type":"ContainerStarted","Data":"d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a"} Nov 03 03:48:28.953631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.953606 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:48:28.954001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.953979 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:48:28.954877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:28.954826 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:48:29.065199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.065143 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" podStartSLOduration=2.06512328 podStartE2EDuration="2.06512328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:29.063062379 +0000 UTC m=+2800.215750800" watchObservedRunningTime="2026-11-03 03:48:29.06512328 +0000 UTC m=+2800.217811702" Nov 03 03:48:29.112686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.112661 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:29.193356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.193324 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\" (UniqueName: \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\") pod \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " Nov 03 03:48:29.193896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.193380 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\") pod \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " Nov 03 03:48:29.193896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.193496 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\") pod \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\" (UID: \"6a7988fb-b018-4c68-a9aa-f392e92ff16c\") " Nov 03 03:48:29.194546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.194425 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca" pod "6a7988fb-b018-4c68-a9aa-f392e92ff16c" (UID: "6a7988fb-b018-4c68-a9aa-f392e92ff16c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:29.196162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.196138 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47" pod "6a7988fb-b018-4c68-a9aa-f392e92ff16c" (UID: "6a7988fb-b018-4c68-a9aa-f392e92ff16c"). InnerVolumeSpecName "kube-api-access-vbt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:29.196982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.196951 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca" pod "6a7988fb-b018-4c68-a9aa-f392e92ff16c" (UID: "6a7988fb-b018-4c68-a9aa-f392e92ff16c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:29.295082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.295048 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbt47\" (UniqueName: \"kubernetes.io/projected/6a7988fb-b018-4c68-a9aa-f392e92ff16c-kube-api-access-vbt47\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:29.295082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.295078 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:29.295082 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.295087 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7988fb-b018-4c68-a9aa-f392e92ff16c-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:29.956186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.956146 5275 generic.go:356] "Generic (PLEG): container finished" podID="f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" containerID="c80f8cff525cb02d529c8800111e15eb6c30ce9677d6b5322936004e1e1a9bf1" exitCode=0 Nov 03 03:48:29.956367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.956289 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" event={"ID":"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3","Type":"ContainerDied","Data":"c80f8cff525cb02d529c8800111e15eb6c30ce9677d6b5322936004e1e1a9bf1"} Nov 03 03:48:29.957785 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.957758 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" Nov 03 03:48:29.957946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.957787 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv" event={"ID":"6a7988fb-b018-4c68-a9aa-f392e92ff16c","Type":"ContainerDied","Data":"14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123"} Nov 03 03:48:29.957946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:29.957817 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e0100680d0ca5ab9886095bb81952825c45c5c300b4db2619ad781b9c36123" Nov 03 03:48:31.120161 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.120133 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:31.220385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.220298 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\" (UniqueName: \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\") pod \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " Nov 03 03:48:31.220530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.220428 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\") pod \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " Nov 03 03:48:31.220530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.220490 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\") pod \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\" (UID: \"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3\") " Nov 03 03:48:31.220935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.220901 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca" pod "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" (UID: "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:31.221083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.221068 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:31.222663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.222637 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca" pod "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" (UID: "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:31.222663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.222650 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl" pod "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" (UID: "f8b2bb35-55e9-4380-a73b-fd3f4ce541b3"). InnerVolumeSpecName "kube-api-access-7m6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:31.322675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.322636 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:31.322675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.322676 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m6gl\" (UniqueName: \"kubernetes.io/projected/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3-kube-api-access-7m6gl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:31.973995 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.973965 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" Nov 03 03:48:31.974180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.973994 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms" event={"ID":"f8b2bb35-55e9-4380-a73b-fd3f4ce541b3","Type":"ContainerDied","Data":"d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a"} Nov 03 03:48:31.974180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:31.974046 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c20c1568ac517385eaa31cbc221a2bd190f3b19503ef3bd479bedd79aaf10a" Nov 03 03:48:32.333135 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:48:32.333090 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:48:37.936776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:37.936746 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:48:37.937325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:37.936819 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:48:41.000183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000143 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd"] Nov 03 03:48:41.000690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000677 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6a7988fb-b018-4c68-a9aa-f392e92ff16c" containerName="revoke-keys" Nov 03 03:48:41.000737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000691 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7988fb-b018-4c68-a9aa-f392e92ff16c" containerName="revoke-keys" Nov 03 03:48:41.000737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000714 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" containerName="revoke-keys" Nov 03 03:48:41.000737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000720 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" containerName="revoke-keys" Nov 03 03:48:41.000833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000813 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7988fb-b018-4c68-a9aa-f392e92ff16c" containerName="revoke-keys" Nov 03 03:48:41.000833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.000824 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" containerName="revoke-keys" Nov 03 03:48:41.004260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.004235 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.055211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.055177 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q964k\" (UniqueName: \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.055428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.055367 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.055500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.055431 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.057945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.057923 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:48:41.062652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.062620 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd"] Nov 03 03:48:41.107545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.107512 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:48:41.107710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.107665 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:48:41.107710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.107692 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:48:41.156500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.156461 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q964k\" (UniqueName: \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.156675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.156523 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.156675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.156563 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.157218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.157188 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.159120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.159105 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.217607 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.217574 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q964k\" (UniqueName: \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\") pod \"maas-api-revoke-sub-2de19551-79gxd\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.318250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.318217 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:41.698546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:41.698509 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd"] Nov 03 03:48:41.699740 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:41.699711 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b726bee_ee6c_415c_87c8_2214cf7e8ce3.slice/crio-8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971 WatchSource:0}: Error finding container 8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971: Status 404 returned error can't find the container with id 8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971 Nov 03 03:48:42.051220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:42.051186 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" event={"ID":"9b726bee-ee6c-415c-87c8-2214cf7e8ce3","Type":"ContainerStarted","Data":"544ea8c33a8e6efa3bbf8fd61d766d22e6146de10590e6e13c625181358a8d67"} Nov 03 03:48:42.051637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:42.051235 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" event={"ID":"9b726bee-ee6c-415c-87c8-2214cf7e8ce3","Type":"ContainerStarted","Data":"8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971"} Nov 03 03:48:42.170061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:42.169993 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" podStartSLOduration=2.169973376 podStartE2EDuration="2.169973376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:42.166566514 +0000 UTC m=+2813.319254935" watchObservedRunningTime="2026-11-03 03:48:42.169973376 +0000 UTC m=+2813.322661803" Nov 03 03:48:43.061440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.061403 5275 generic.go:356] "Generic (PLEG): container finished" podID="9b726bee-ee6c-415c-87c8-2214cf7e8ce3" containerID="544ea8c33a8e6efa3bbf8fd61d766d22e6146de10590e6e13c625181358a8d67" exitCode=0 Nov 03 03:48:43.061902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.061457 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" event={"ID":"9b726bee-ee6c-415c-87c8-2214cf7e8ce3","Type":"ContainerDied","Data":"544ea8c33a8e6efa3bbf8fd61d766d22e6146de10590e6e13c625181358a8d67"} Nov 03 03:48:43.773769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.773726 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt"] Nov 03 03:48:43.777883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.777863 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.790450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.790424 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8vz\" (UniqueName: \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.790595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.790556 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.790745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.790726 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.859814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.859771 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt"] Nov 03 03:48:43.892093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.892048 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.892312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.892191 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.892312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.892270 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8vz\" (UniqueName: \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.892816 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.892794 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.895455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.895428 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:43.962602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:43.962572 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8vz\" (UniqueName: \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\") pod \"maas-api-revoke-sub-27f688b6-pwkdt\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:44.092199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.092094 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:44.296690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.296658 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:44.400028 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.399995 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\" (UniqueName: \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\") pod \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " Nov 03 03:48:44.400217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.400091 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\") pod \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " Nov 03 03:48:44.400217 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.400139 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\") pod \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\" (UID: \"9b726bee-ee6c-415c-87c8-2214cf7e8ce3\") " Nov 03 03:48:44.400561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.400528 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca" pod "9b726bee-ee6c-415c-87c8-2214cf7e8ce3" (UID: "9b726bee-ee6c-415c-87c8-2214cf7e8ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:44.402243 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.402221 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k" pod "9b726bee-ee6c-415c-87c8-2214cf7e8ce3" (UID: "9b726bee-ee6c-415c-87c8-2214cf7e8ce3"). InnerVolumeSpecName "kube-api-access-q964k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:44.402352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.402256 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca" pod "9b726bee-ee6c-415c-87c8-2214cf7e8ce3" (UID: "9b726bee-ee6c-415c-87c8-2214cf7e8ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:44.501452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.501418 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q964k\" (UniqueName: \"kubernetes.io/projected/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-kube-api-access-q964k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:44.501452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.501449 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:44.501452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.501458 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b726bee-ee6c-415c-87c8-2214cf7e8ce3-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:44.528544 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:44.528516 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt"] Nov 03 03:48:44.529482 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:48:44.529451 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4c3fdb_662b_4d08_88d9_379866352e34.slice/crio-1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de WatchSource:0}: Error finding container 1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de: Status 404 returned error can't find the container with id 1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de Nov 03 03:48:45.111012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.110902 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" event={"ID":"0d4c3fdb-662b-4d08-88d9-379866352e34","Type":"ContainerStarted","Data":"073d17fb2145c9fc5fb08ab04f288be758b76ae3f4afbc899111ed8db1602456"} Nov 03 03:48:45.111012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.110992 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" event={"ID":"0d4c3fdb-662b-4d08-88d9-379866352e34","Type":"ContainerStarted","Data":"1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de"} Nov 03 03:48:45.112485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.112455 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" event={"ID":"9b726bee-ee6c-415c-87c8-2214cf7e8ce3","Type":"ContainerDied","Data":"8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971"} Nov 03 03:48:45.112627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.112497 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd" Nov 03 03:48:45.112627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.112503 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cc181ddbae91c9a4b674d8245a47e4cae473e7d3e63e07d39124b976bf48971" Nov 03 03:48:45.244240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:45.244108 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" podStartSLOduration=2.244088031 podStartE2EDuration="2.244088031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:45.233737323 +0000 UTC m=+2816.386425742" watchObservedRunningTime="2026-11-03 03:48:45.244088031 +0000 UTC m=+2816.396776470" Nov 03 03:48:46.121264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:46.121229 5275 generic.go:356] "Generic (PLEG): container finished" podID="0d4c3fdb-662b-4d08-88d9-379866352e34" containerID="073d17fb2145c9fc5fb08ab04f288be758b76ae3f4afbc899111ed8db1602456" exitCode=0 Nov 03 03:48:46.121764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:46.121310 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" event={"ID":"0d4c3fdb-662b-4d08-88d9-379866352e34","Type":"ContainerDied","Data":"073d17fb2145c9fc5fb08ab04f288be758b76ae3f4afbc899111ed8db1602456"} Nov 03 03:48:47.274263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.274237 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:47.336704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.336667 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\") pod \"0d4c3fdb-662b-4d08-88d9-379866352e34\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " Nov 03 03:48:47.336921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.336837 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\") pod \"0d4c3fdb-662b-4d08-88d9-379866352e34\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " Nov 03 03:48:47.336998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.336969 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\" (UniqueName: \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\") pod \"0d4c3fdb-662b-4d08-88d9-379866352e34\" (UID: \"0d4c3fdb-662b-4d08-88d9-379866352e34\") " Nov 03 03:48:47.337181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.337158 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca" pod "0d4c3fdb-662b-4d08-88d9-379866352e34" (UID: "0d4c3fdb-662b-4d08-88d9-379866352e34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:47.337409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.337391 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:47.339007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.338982 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca" pod "0d4c3fdb-662b-4d08-88d9-379866352e34" (UID: "0d4c3fdb-662b-4d08-88d9-379866352e34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:47.339086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.339076 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz" pod "0d4c3fdb-662b-4d08-88d9-379866352e34" (UID: "0d4c3fdb-662b-4d08-88d9-379866352e34"). InnerVolumeSpecName "kube-api-access-8q8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:47.438797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.438709 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8vz\" (UniqueName: \"kubernetes.io/projected/0d4c3fdb-662b-4d08-88d9-379866352e34-kube-api-access-8q8vz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:47.438797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:47.438740 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c3fdb-662b-4d08-88d9-379866352e34-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:48.136994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:48.136949 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" event={"ID":"0d4c3fdb-662b-4d08-88d9-379866352e34","Type":"ContainerDied","Data":"1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de"} Nov 03 03:48:48.137191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:48.137000 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt" Nov 03 03:48:48.137191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:48.137004 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a017b9aacad9b75b506708833a86adcfa79252c89002dbbad85bb69a92064de" Nov 03 03:48:54.573588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.573549 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c"] Nov 03 03:48:54.574354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574330 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0d4c3fdb-662b-4d08-88d9-379866352e34" containerName="revoke-keys" Nov 03 03:48:54.574453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574356 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4c3fdb-662b-4d08-88d9-379866352e34" containerName="revoke-keys" Nov 03 03:48:54.574453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574384 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9b726bee-ee6c-415c-87c8-2214cf7e8ce3" containerName="revoke-keys" Nov 03 03:48:54.574453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574392 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b726bee-ee6c-415c-87c8-2214cf7e8ce3" containerName="revoke-keys" Nov 03 03:48:54.574611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574578 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4c3fdb-662b-4d08-88d9-379866352e34" containerName="revoke-keys" Nov 03 03:48:54.574611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.574592 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b726bee-ee6c-415c-87c8-2214cf7e8ce3" containerName="revoke-keys" Nov 03 03:48:54.579360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.579335 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.632526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.632477 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:48:54.636280 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.636242 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvsc\" (UniqueName: \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.636446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.636428 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.636638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.636615 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.682195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.682158 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:48:54.682195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.682192 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:48:54.682471 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.682292 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:48:54.686409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.686378 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c"] Nov 03 03:48:54.737472 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.737432 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.737656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.737566 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.737656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.737618 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvsc\" (UniqueName: \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.738216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.738190 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.740128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.740106 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.807050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.807005 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvsc\" (UniqueName: \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\") pod \"maas-api-revoke-sub-37b9d4b5-skk6c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:54.893830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:54.893731 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:55.286112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:55.284942 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c"] Nov 03 03:48:56.232932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:56.232893 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" event={"ID":"3421e84e-eed0-4638-bc7b-8ed59148999c","Type":"ContainerStarted","Data":"8e8be5b849751317c91e83f8fb590b7873808f8b542797192f10294e7a734f13"} Nov 03 03:48:56.233415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:56.232990 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" event={"ID":"3421e84e-eed0-4638-bc7b-8ed59148999c","Type":"ContainerStarted","Data":"1213770eb7d6c967769e36276742754681e098abd1fea328dc5cc9195acb7858"} Nov 03 03:48:56.372094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:56.372032 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" podStartSLOduration=2.372012576 podStartE2EDuration="2.372012576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:56.348051024 +0000 UTC m=+2827.500739445" watchObservedRunningTime="2026-11-03 03:48:56.372012576 +0000 UTC m=+2827.524700998" Nov 03 03:48:57.242630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:57.242594 5275 generic.go:356] "Generic (PLEG): container finished" podID="3421e84e-eed0-4638-bc7b-8ed59148999c" containerID="8e8be5b849751317c91e83f8fb590b7873808f8b542797192f10294e7a734f13" exitCode=0 Nov 03 03:48:57.243148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:57.242691 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" event={"ID":"3421e84e-eed0-4638-bc7b-8ed59148999c","Type":"ContainerDied","Data":"8e8be5b849751317c91e83f8fb590b7873808f8b542797192f10294e7a734f13"} Nov 03 03:48:58.404446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.404420 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:58.515290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.515197 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\") pod \"3421e84e-eed0-4638-bc7b-8ed59148999c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " Nov 03 03:48:58.515477 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.515307 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\" (UniqueName: \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\") pod \"3421e84e-eed0-4638-bc7b-8ed59148999c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " Nov 03 03:48:58.515541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.515491 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\") pod \"3421e84e-eed0-4638-bc7b-8ed59148999c\" (UID: \"3421e84e-eed0-4638-bc7b-8ed59148999c\") " Nov 03 03:48:58.515595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.515553 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca" pod "3421e84e-eed0-4638-bc7b-8ed59148999c" (UID: "3421e84e-eed0-4638-bc7b-8ed59148999c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:48:58.515936 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.515919 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:58.517591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.517563 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca" pod "3421e84e-eed0-4638-bc7b-8ed59148999c" (UID: "3421e84e-eed0-4638-bc7b-8ed59148999c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:48:58.517710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.517600 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc" pod "3421e84e-eed0-4638-bc7b-8ed59148999c" (UID: "3421e84e-eed0-4638-bc7b-8ed59148999c"). InnerVolumeSpecName "kube-api-access-6dvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:48:58.617184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.617146 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3421e84e-eed0-4638-bc7b-8ed59148999c-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:58.617184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:58.617178 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dvsc\" (UniqueName: \"kubernetes.io/projected/3421e84e-eed0-4638-bc7b-8ed59148999c-kube-api-access-6dvsc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:48:59.259442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:59.259366 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" Nov 03 03:48:59.259442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:59.259380 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c" event={"ID":"3421e84e-eed0-4638-bc7b-8ed59148999c","Type":"ContainerDied","Data":"1213770eb7d6c967769e36276742754681e098abd1fea328dc5cc9195acb7858"} Nov 03 03:48:59.259442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:48:59.259420 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1213770eb7d6c967769e36276742754681e098abd1fea328dc5cc9195acb7858" Nov 03 03:49:02.329666 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:49:02.329630 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:49:02.873283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.873254 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn"] Nov 03 03:49:02.873793 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.873781 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3421e84e-eed0-4638-bc7b-8ed59148999c" containerName="revoke-keys" Nov 03 03:49:02.873840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.873795 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3421e84e-eed0-4638-bc7b-8ed59148999c" containerName="revoke-keys" Nov 03 03:49:02.873944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.873934 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3421e84e-eed0-4638-bc7b-8ed59148999c" containerName="revoke-keys" Nov 03 03:49:02.878837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.878810 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:02.931044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.931009 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:49:02.935725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.935692 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn"] Nov 03 03:49:02.968555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.968510 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:02.968744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.968595 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gng\" (UniqueName: \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:02.968744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.968632 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:02.980580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.980556 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:49:02.980580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.980576 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:49:02.980787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:02.980590 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:49:03.070421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.070382 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.070616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.070498 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gng\" (UniqueName: \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.070616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.070544 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.071190 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.071169 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.073093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.073071 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.132001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.131923 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gng\" (UniqueName: \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\") pod \"maas-api-revoke-sub-dae46d00-mhdmn\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.193364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.193332 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:03.574451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:03.574418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn"] Nov 03 03:49:03.574822 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:49:03.574605 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0800e6_2d04_4be0_a325_c967caaeec10.slice/crio-cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568 WatchSource:0}: Error finding container cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568: Status 404 returned error can't find the container with id cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568 Nov 03 03:49:04.301044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:04.301005 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" event={"ID":"fb0800e6-2d04-4be0-a325-c967caaeec10","Type":"ContainerStarted","Data":"2a81bdb386c653c52eae995d62fbc305205fa4a681d7585dd6efc2c2391f81a5"} Nov 03 03:49:04.301266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:04.301071 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" event={"ID":"fb0800e6-2d04-4be0-a325-c967caaeec10","Type":"ContainerStarted","Data":"cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568"} Nov 03 03:49:04.418537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:04.418479 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" podStartSLOduration=2.418463109 podStartE2EDuration="2.418463109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:04.416265912 +0000 UTC m=+2835.568954333" watchObservedRunningTime="2026-11-03 03:49:04.418463109 +0000 UTC m=+2835.571151529" Nov 03 03:49:05.309731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:05.309693 5275 generic.go:356] "Generic (PLEG): container finished" podID="fb0800e6-2d04-4be0-a325-c967caaeec10" containerID="2a81bdb386c653c52eae995d62fbc305205fa4a681d7585dd6efc2c2391f81a5" exitCode=0 Nov 03 03:49:05.310153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:05.309774 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" event={"ID":"fb0800e6-2d04-4be0-a325-c967caaeec10","Type":"ContainerDied","Data":"2a81bdb386c653c52eae995d62fbc305205fa4a681d7585dd6efc2c2391f81a5"} Nov 03 03:49:06.485603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.485575 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:06.509819 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.509791 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\" (UniqueName: \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\") pod \"fb0800e6-2d04-4be0-a325-c967caaeec10\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " Nov 03 03:49:06.510030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.509858 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\") pod \"fb0800e6-2d04-4be0-a325-c967caaeec10\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " Nov 03 03:49:06.510030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.509900 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\") pod \"fb0800e6-2d04-4be0-a325-c967caaeec10\" (UID: \"fb0800e6-2d04-4be0-a325-c967caaeec10\") " Nov 03 03:49:06.510343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.510309 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca" pod "fb0800e6-2d04-4be0-a325-c967caaeec10" (UID: "fb0800e6-2d04-4be0-a325-c967caaeec10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:06.512096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.512062 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca" pod "fb0800e6-2d04-4be0-a325-c967caaeec10" (UID: "fb0800e6-2d04-4be0-a325-c967caaeec10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:06.512096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.512072 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng" pod "fb0800e6-2d04-4be0-a325-c967caaeec10" (UID: "fb0800e6-2d04-4be0-a325-c967caaeec10"). InnerVolumeSpecName "kube-api-access-w9gng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:06.612064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.611984 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gng\" (UniqueName: \"kubernetes.io/projected/fb0800e6-2d04-4be0-a325-c967caaeec10-kube-api-access-w9gng\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:06.612064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.612022 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:06.612064 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:06.612037 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0800e6-2d04-4be0-a325-c967caaeec10-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:07.326272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:07.326236 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" Nov 03 03:49:07.326272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:07.326252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn" event={"ID":"fb0800e6-2d04-4be0-a325-c967caaeec10","Type":"ContainerDied","Data":"cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568"} Nov 03 03:49:07.326564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:07.326313 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb3836c2731f3ea3a92d687e5ad737ca1735d996089380f915cd74be9544f568" Nov 03 03:49:15.470790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.470754 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:49:15.471359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.471345 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fb0800e6-2d04-4be0-a325-c967caaeec10" containerName="revoke-keys" Nov 03 03:49:15.471406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.471360 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0800e6-2d04-4be0-a325-c967caaeec10" containerName="revoke-keys" Nov 03 03:49:15.471474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.471440 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0800e6-2d04-4be0-a325-c967caaeec10" containerName="revoke-keys" Nov 03 03:49:15.475177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.475150 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527452 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlm8q\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527519 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527652 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527623 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527668 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527749 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527825 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.527980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.527964 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.528019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.528009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.528067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.528044 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.584546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.584505 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:49:15.629669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629632 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.629895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629682 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.629895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629704 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.629895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629728 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.629895 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629824 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629904 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.629935 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630013 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlm8q\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630065 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630096 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630145 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630182 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630297 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.630673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.630655 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.632299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.632279 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.632501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.632481 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.690229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.690190 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.690552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.690522 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlm8q\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:15.792306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:15.792269 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:16.234457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.234423 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:49:16.234665 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:49:16.234640 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c8d3bc_75c6_46a0_bfd5_c458128904de.slice/crio-15350d22e6146ba46ffb56822b5e5b00b6b72567e92fcc379b6a3f9fd1811cde WatchSource:0}: Error finding container 15350d22e6146ba46ffb56822b5e5b00b6b72567e92fcc379b6a3f9fd1811cde: Status 404 returned error can't find the container with id 15350d22e6146ba46ffb56822b5e5b00b6b72567e92fcc379b6a3f9fd1811cde Nov 03 03:49:16.237244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.237208 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:16.237338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.237272 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:16.237338 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.237301 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:16.403180 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.403139 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" event={"ID":"02c8d3bc-75c6-46a0-bfd5-c458128904de","Type":"ContainerStarted","Data":"e87cf69df2e35202a6adcf7d6b435b33d95c417b15cf7af7cdea6d170e0ba9f9"} Nov 03 03:49:16.403340 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.403211 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" event={"ID":"02c8d3bc-75c6-46a0-bfd5-c458128904de","Type":"ContainerStarted","Data":"15350d22e6146ba46ffb56822b5e5b00b6b72567e92fcc379b6a3f9fd1811cde"} Nov 03 03:49:16.527019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.526971 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podStartSLOduration=1.526955988 podStartE2EDuration="1.526955988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:16.522761265 +0000 UTC m=+2847.675449687" watchObservedRunningTime="2026-11-03 03:49:16.526955988 +0000 UTC m=+2847.679644499" Nov 03 03:49:16.545941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.545905 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt"] Nov 03 03:49:16.550341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.550313 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.605175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.605093 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:49:16.605175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.605099 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:49:16.605175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.605146 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:49:16.606275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.605706 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:49:16.610649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.610624 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt"] Nov 03 03:49:16.643239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.643201 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvc4\" (UniqueName: \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.643453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.643296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.643453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.643377 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.745090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.745057 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvc4\" (UniqueName: \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.745288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.745125 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.745288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.745205 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.745903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.745877 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.747663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.747637 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.792424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.792334 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:16.806386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.806357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvc4\" (UniqueName: \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\") pod \"maas-api-revoke-sub-1a43c57e-qb8pt\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:16.868284 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:16.868245 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:17.195601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.195515 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:49:17.196088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.196049 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" containerID="cri-o://5b97cce412124a90c63c0960f0dfa2234c10d381163e9ace6fe508db79082dbc" gracePeriod=30 Nov 03 03:49:17.246566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.246534 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt"] Nov 03 03:49:17.247101 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:49:17.247076 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ad702d_04f1_4887_98fe_e3db19178a95.slice/crio-b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3 WatchSource:0}: Error finding container b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3: Status 404 returned error can't find the container with id b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3 Nov 03 03:49:17.412922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.412696 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" event={"ID":"33ad702d-04f1-4887-98fe-e3db19178a95","Type":"ContainerStarted","Data":"b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3"} Nov 03 03:49:17.796027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.794960 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:17.796027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:17.795018 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:18.421784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:18.421750 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" event={"ID":"33ad702d-04f1-4887-98fe-e3db19178a95","Type":"ContainerStarted","Data":"acb65241bdbbe72f0a08033fced0c15554f088efe8a903906ccc7047bd1dc7b2"} Nov 03 03:49:18.545633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:18.545580 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" podStartSLOduration=2.5455646720000003 podStartE2EDuration="2.545564672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:18.543459886 +0000 UTC m=+2849.696148305" watchObservedRunningTime="2026-11-03 03:49:18.545564672 +0000 UTC m=+2849.698253092" Nov 03 03:49:18.793651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:18.793613 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:18.793876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:18.793680 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:19.430980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:19.430944 5275 generic.go:356] "Generic (PLEG): container finished" podID="33ad702d-04f1-4887-98fe-e3db19178a95" containerID="acb65241bdbbe72f0a08033fced0c15554f088efe8a903906ccc7047bd1dc7b2" exitCode=0 Nov 03 03:49:19.431418 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:19.430983 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" event={"ID":"33ad702d-04f1-4887-98fe-e3db19178a95","Type":"ContainerDied","Data":"acb65241bdbbe72f0a08033fced0c15554f088efe8a903906ccc7047bd1dc7b2"} Nov 03 03:49:19.793384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:19.793344 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:19.793590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:19.793407 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:20.675994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.675964 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:20.793660 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.793627 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:20.793860 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.793685 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:20.827106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.827065 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\") pod \"33ad702d-04f1-4887-98fe-e3db19178a95\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " Nov 03 03:49:20.828708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.828679 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\") pod \"33ad702d-04f1-4887-98fe-e3db19178a95\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " Nov 03 03:49:20.829073 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.829048 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca" pod "33ad702d-04f1-4887-98fe-e3db19178a95" (UID: "33ad702d-04f1-4887-98fe-e3db19178a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:20.829298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.829280 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\" (UniqueName: \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\") pod \"33ad702d-04f1-4887-98fe-e3db19178a95\" (UID: \"33ad702d-04f1-4887-98fe-e3db19178a95\") " Nov 03 03:49:20.829833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.829812 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:20.832266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.832238 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca" pod "33ad702d-04f1-4887-98fe-e3db19178a95" (UID: "33ad702d-04f1-4887-98fe-e3db19178a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:20.838199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.838175 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4" pod "33ad702d-04f1-4887-98fe-e3db19178a95" (UID: "33ad702d-04f1-4887-98fe-e3db19178a95"). InnerVolumeSpecName "kube-api-access-4jvc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:20.931370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.931332 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvc4\" (UniqueName: \"kubernetes.io/projected/33ad702d-04f1-4887-98fe-e3db19178a95-kube-api-access-4jvc4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:20.931370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:20.931363 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ad702d-04f1-4887-98fe-e3db19178a95-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:21.208788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.208706 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf"] Nov 03 03:49:21.209371 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.209355 5275 cpu_manager.go:414] "containerMap: removing container" podUID="33ad702d-04f1-4887-98fe-e3db19178a95" containerName="revoke-keys" Nov 03 03:49:21.209429 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.209373 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ad702d-04f1-4887-98fe-e3db19178a95" containerName="revoke-keys" Nov 03 03:49:21.209485 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.209468 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ad702d-04f1-4887-98fe-e3db19178a95" containerName="revoke-keys" Nov 03 03:49:21.213121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.213099 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.283656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.283595 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf"] Nov 03 03:49:21.335592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.335557 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.335770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.335622 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtph\" (UniqueName: \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.335821 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.335797 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.436949 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.436917 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.437166 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.437049 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.437166 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.437103 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtph\" (UniqueName: \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.437685 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.437658 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.439567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.439546 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.465929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.463572 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" event={"ID":"33ad702d-04f1-4887-98fe-e3db19178a95","Type":"ContainerDied","Data":"b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3"} Nov 03 03:49:21.465929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.463642 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6ca5b21eee516eb054445b7012c91e33c23106223642cb49eadcff7bc713ba3" Nov 03 03:49:21.465929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.463715 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt" Nov 03 03:49:21.498399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.498368 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtph\" (UniqueName: \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\") pod \"maas-api-revoke-sub-5e385438-m7lgf\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.526910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.526878 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:21.793122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.793082 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:21.793548 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.793139 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:21.932747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:21.932722 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf"] Nov 03 03:49:21.935155 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:49:21.935127 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c791978_64cf_4803_8dac_300897e4d032.slice/crio-1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20 WatchSource:0}: Error finding container 1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20: Status 404 returned error can't find the container with id 1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20 Nov 03 03:49:22.504915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.496641 5275 generic.go:356] "Generic (PLEG): container finished" podID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerID="5b97cce412124a90c63c0960f0dfa2234c10d381163e9ace6fe508db79082dbc" exitCode=0 Nov 03 03:49:22.504915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.496757 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerDied","Data":"5b97cce412124a90c63c0960f0dfa2234c10d381163e9ace6fe508db79082dbc"} Nov 03 03:49:22.504915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.496813 5275 scope.go:122] "RemoveContainer" containerID="d11348047229602a94be7df2844e3bd4b3a0abb96a4cc18b2ba814758edf652f" Nov 03 03:49:22.509496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.509460 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" event={"ID":"7c791978-64cf-4803-8dac-300897e4d032","Type":"ContainerStarted","Data":"6c4061bd27838ef7e0c4614b0eb5515be4b11581c3682c3db31d374cc66d4fde"} Nov 03 03:49:22.509619 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.509526 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" event={"ID":"7c791978-64cf-4803-8dac-300897e4d032","Type":"ContainerStarted","Data":"1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20"} Nov 03 03:49:22.521496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.521468 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:49:22.642379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.642332 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" podStartSLOduration=1.6423170040000001 podStartE2EDuration="1.642317004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:22.638761247 +0000 UTC m=+2853.791449692" watchObservedRunningTime="2026-11-03 03:49:22.642317004 +0000 UTC m=+2853.795005424" Nov 03 03:49:22.662878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.662125 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.662935 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663020 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663083 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663126 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663195 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663277 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663308 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663357 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\") pod \"c6babc80-bdb9-459b-be16-991b3f4a2380\" (UID: \"c6babc80-bdb9-459b-be16-991b3f4a2380\") " Nov 03 03:49:22.663893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.663791 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:22.664331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.664071 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.664528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.664502 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:22.665001 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.664971 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:22.665100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.665019 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:22.665483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.665448 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:22.666141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.666116 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:22.666505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.666479 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:22.666790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.666774 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:49:22.666989 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.666972 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w" pod "c6babc80-bdb9-459b-be16-991b3f4a2380" (UID: "c6babc80-bdb9-459b-be16-991b3f4a2380"). InnerVolumeSpecName "kube-api-access-nbg8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:22.765503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765420 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbg8w\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-kube-api-access-nbg8w\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765458 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6babc80-bdb9-459b-be16-991b3f4a2380-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765476 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765490 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765504 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765517 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6babc80-bdb9-459b-be16-991b3f4a2380-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765529 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.765940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.765543 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6babc80-bdb9-459b-be16-991b3f4a2380-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:22.793083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.793046 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 03 03:49:22.793275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:22.793113 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 03 03:49:23.521474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.521449 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" Nov 03 03:49:23.521663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.521445 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2" event={"ID":"c6babc80-bdb9-459b-be16-991b3f4a2380","Type":"ContainerDied","Data":"eafd5cb965485bce8f1c5417d427cc338eb2b74baae3e4e67f06d5843a99901f"} Nov 03 03:49:23.521663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.521593 5275 scope.go:122] "RemoveContainer" containerID="5b97cce412124a90c63c0960f0dfa2234c10d381163e9ace6fe508db79082dbc" Nov 03 03:49:23.523756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.523677 5275 generic.go:356] "Generic (PLEG): container finished" podID="7c791978-64cf-4803-8dac-300897e4d032" containerID="6c4061bd27838ef7e0c4614b0eb5515be4b11581c3682c3db31d374cc66d4fde" exitCode=0 Nov 03 03:49:23.523756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.523717 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" event={"ID":"7c791978-64cf-4803-8dac-300897e4d032","Type":"ContainerDied","Data":"6c4061bd27838ef7e0c4614b0eb5515be4b11581c3682c3db31d374cc66d4fde"} Nov 03 03:49:23.875226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.875133 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:49:23.880783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:23.880757 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2be7ec-openshift-default-678fd77985-r42m2"] Nov 03 03:49:24.058708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.058677 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:24.060154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.060003 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:24.062426 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.062405 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:49:24.688256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.688227 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:24.791115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.791022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\") pod \"7c791978-64cf-4803-8dac-300897e4d032\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " Nov 03 03:49:24.791115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.791100 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\") pod \"7c791978-64cf-4803-8dac-300897e4d032\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " Nov 03 03:49:24.791354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.791169 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\" (UniqueName: \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\") pod \"7c791978-64cf-4803-8dac-300897e4d032\" (UID: \"7c791978-64cf-4803-8dac-300897e4d032\") " Nov 03 03:49:24.791468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.791442 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca" pod "7c791978-64cf-4803-8dac-300897e4d032" (UID: "7c791978-64cf-4803-8dac-300897e4d032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:24.791643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.791622 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c791978-64cf-4803-8dac-300897e4d032-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:24.793404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.793371 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca" pod "7c791978-64cf-4803-8dac-300897e4d032" (UID: "7c791978-64cf-4803-8dac-300897e4d032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:24.793494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.793445 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph" pod "7c791978-64cf-4803-8dac-300897e4d032" (UID: "7c791978-64cf-4803-8dac-300897e4d032"). InnerVolumeSpecName "kube-api-access-vxtph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:24.893229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.893182 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtph\" (UniqueName: \"kubernetes.io/projected/7c791978-64cf-4803-8dac-300897e4d032-kube-api-access-vxtph\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:24.893229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.893218 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c791978-64cf-4803-8dac-300897e4d032-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:24.985476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:24.985436 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" path="/var/lib/kubelet/pods/c6babc80-bdb9-459b-be16-991b3f4a2380/volumes" Nov 03 03:49:25.540317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:25.540289 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" Nov 03 03:49:25.540505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:25.540328 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf" event={"ID":"7c791978-64cf-4803-8dac-300897e4d032","Type":"ContainerDied","Data":"1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20"} Nov 03 03:49:25.540505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:25.540384 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd92429ff08f02abba9ebf0fbd886b92a1b4be8d6b8d4f3a6d917934b95bd20" Nov 03 03:49:32.329888 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:49:32.329834 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:49:37.948187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.948152 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:49:37.948754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.948737 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7c791978-64cf-4803-8dac-300897e4d032" containerName="revoke-keys" Nov 03 03:49:37.948754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.948753 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c791978-64cf-4803-8dac-300897e4d032" containerName="revoke-keys" Nov 03 03:49:37.948926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.948765 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:37.948926 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.948770 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:37.950002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.949965 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c791978-64cf-4803-8dac-300897e4d032" containerName="revoke-keys" Nov 03 03:49:37.950002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.949996 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:37.955885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:37.955831 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.078987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.078951 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:49:38.085323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.085289 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.085648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.085621 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrdf\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.085823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.085804 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.086006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.085987 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.089094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.089054 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.089318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.089283 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.093663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.093638 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.093947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.093912 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.094163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.094123 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.196655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.196609 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrdf\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.206081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.205985 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215076 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215168 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215227 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215306 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215413 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215474 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.215546 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.216373 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.216701 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.217881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.217004 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.218609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.218116 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.218609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.218448 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.220912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.219985 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.220912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.220706 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.309903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.309468 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.314620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.314584 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrdf\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:38.600183 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:38.600145 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:39.313658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.313618 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:39.314105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.313687 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:39.314105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.313716 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:39.317081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.315568 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:49:39.708515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.708430 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" event={"ID":"e5104b7b-3513-4c05-9afe-a77545cc7b9f","Type":"ContainerStarted","Data":"b0494fe516652a7b10755c8173e9072cf4532a8568c5943fedbb69e6d8864a4a"} Nov 03 03:49:39.708515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.708508 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" event={"ID":"e5104b7b-3513-4c05-9afe-a77545cc7b9f","Type":"ContainerStarted","Data":"db4ea4f6ad9d3a71e98a0f4865740d2a36f4890a7cd87d8ad4dcc10cf8718189"} Nov 03 03:49:39.843529 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:39.843472 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podStartSLOduration=2.843458119 podStartE2EDuration="2.843458119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:39.839571202 +0000 UTC m=+2870.992259618" watchObservedRunningTime="2026-11-03 03:49:39.843458119 +0000 UTC m=+2870.996146606" Nov 03 03:49:40.435764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.435727 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d"] Nov 03 03:49:40.436392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.436376 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:40.436452 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.436393 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:40.436510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.436501 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6babc80-bdb9-459b-be16-991b3f4a2380" containerName="istio-proxy" Nov 03 03:49:40.440056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.440041 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.474034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.473633 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.474034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.473866 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d487d\" (UniqueName: \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.474034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.473908 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.504427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.502891 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:49:40.504427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.503637 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:49:40.514618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.514574 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d"] Nov 03 03:49:40.596874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.590256 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:49:40.604902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.601060 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:49:40.604902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.602139 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d487d\" (UniqueName: \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.604902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.602194 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.604902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.602262 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.605947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.605912 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:40.607533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.607501 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.607896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.607870 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.609324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.609294 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:40.609408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.609361 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:40.691968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.686087 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d487d\" (UniqueName: \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\") pod \"maas-api-revoke-sub-47aca150-6tf5d\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:40.781874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:40.780058 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:41.327740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:41.320025 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d"] Nov 03 03:49:41.610893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:41.603264 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:41.610893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:41.603326 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:41.734362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:41.734320 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" event={"ID":"b409d49f-4957-4fdf-b77c-e5589fbf0882","Type":"ContainerStarted","Data":"c57e5ec9ad1fef262ffa466b52083e24c56de0cf6792c49a43042196c18af302"} Nov 03 03:49:42.601987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:42.601283 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:42.601987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:42.601342 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:42.743898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:42.743866 5275 generic.go:356] "Generic (PLEG): container finished" podID="b409d49f-4957-4fdf-b77c-e5589fbf0882" containerID="8477318d3fdacca6ef4ae15a407c96c31dfeb0308d8896bd0cb70f39e46f37e7" exitCode=0 Nov 03 03:49:42.744302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:42.743953 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" event={"ID":"b409d49f-4957-4fdf-b77c-e5589fbf0882","Type":"ContainerDied","Data":"8477318d3fdacca6ef4ae15a407c96c31dfeb0308d8896bd0cb70f39e46f37e7"} Nov 03 03:49:43.602247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:43.602194 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:43.602451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:43.602280 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:43.984562 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:43.984536 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:44.076352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.076316 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\") pod \"b409d49f-4957-4fdf-b77c-e5589fbf0882\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " Nov 03 03:49:44.076527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.076425 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\") pod \"b409d49f-4957-4fdf-b77c-e5589fbf0882\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " Nov 03 03:49:44.076609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.076593 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\" (UniqueName: \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\") pod \"b409d49f-4957-4fdf-b77c-e5589fbf0882\" (UID: \"b409d49f-4957-4fdf-b77c-e5589fbf0882\") " Nov 03 03:49:44.076901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.076862 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca" pod "b409d49f-4957-4fdf-b77c-e5589fbf0882" (UID: "b409d49f-4957-4fdf-b77c-e5589fbf0882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:44.077318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.077282 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:44.078776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.078749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca" pod "b409d49f-4957-4fdf-b77c-e5589fbf0882" (UID: "b409d49f-4957-4fdf-b77c-e5589fbf0882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:44.079395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.079376 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d" pod "b409d49f-4957-4fdf-b77c-e5589fbf0882" (UID: "b409d49f-4957-4fdf-b77c-e5589fbf0882"). InnerVolumeSpecName "kube-api-access-d487d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:44.185662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.185554 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d487d\" (UniqueName: \"kubernetes.io/projected/b409d49f-4957-4fdf-b77c-e5589fbf0882-kube-api-access-d487d\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:44.185662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.185595 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b409d49f-4957-4fdf-b77c-e5589fbf0882-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:44.601671 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.601623 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:44.601866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.601694 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:44.784708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.784681 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" Nov 03 03:49:44.784909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.784711 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d" event={"ID":"b409d49f-4957-4fdf-b77c-e5589fbf0882","Type":"ContainerDied","Data":"c57e5ec9ad1fef262ffa466b52083e24c56de0cf6792c49a43042196c18af302"} Nov 03 03:49:44.784909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:44.784753 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57e5ec9ad1fef262ffa466b52083e24c56de0cf6792c49a43042196c18af302" Nov 03 03:49:45.601807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:45.601767 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 03:49:45.602446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:45.602403 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 03:49:47.074760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:47.074724 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:47.075417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:47.075108 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:47.076139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:47.076113 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:49:48.817948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.817912 5275 generic.go:356] "Generic (PLEG): container finished" podID="ac72609c-50df-45e2-9b87-d5e426c55446" containerID="d17fec1a99dfddab749d602d7f4a88d22b175135877b41e2f09fc8a111193864" exitCode=0 Nov 03 03:49:48.818406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.817988 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerDied","Data":"d17fec1a99dfddab749d602d7f4a88d22b175135877b41e2f09fc8a111193864"} Nov 03 03:49:48.818406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.818365 5275 scope.go:122] "RemoveContainer" containerID="d17fec1a99dfddab749d602d7f4a88d22b175135877b41e2f09fc8a111193864" Nov 03 03:49:48.819735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.819709 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:48.819814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.819791 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:48.819878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.819824 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:49:48.821575 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:48.821562 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:49:49.826577 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:49.826538 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerStarted","Data":"e64f5d8b09090df22908d6033079218f519de2efffab3a151f2ded2990d83cfa"} Nov 03 03:49:50.612764 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:50.612727 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:49:50.613906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:50.613876 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 03:49:50.614041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:50.613935 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 03:49:51.612312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:51.612275 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 03:49:51.612765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:51.612342 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 03:49:52.612473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:52.612390 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 03:49:52.612473 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:52.612446 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 03:49:53.612547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:53.612507 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 03:49:53.613004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:53.612558 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 03:49:54.612789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:54.612753 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 03 03:49:54.613231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:54.612807 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 03 03:49:55.617467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.617436 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:49:55.617885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.617837 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:49:55.618565 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.618542 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:49:55.772068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.772027 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g"] Nov 03 03:49:55.773057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.773027 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b409d49f-4957-4fdf-b77c-e5589fbf0882" containerName="revoke-keys" Nov 03 03:49:55.773057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.773055 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b409d49f-4957-4fdf-b77c-e5589fbf0882" containerName="revoke-keys" Nov 03 03:49:55.773301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.773219 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b409d49f-4957-4fdf-b77c-e5589fbf0882" containerName="revoke-keys" Nov 03 03:49:55.803871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.803165 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.857205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.857174 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:49:55.858434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.857531 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:49:55.858606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.857550 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:49:55.858691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.857591 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:49:55.858771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.858349 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.858982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.858965 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97k4x\" (UniqueName: \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.859270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.859254 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.861350 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.861327 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g"] Nov 03 03:49:55.960831 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.960739 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.961639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.961610 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.961952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.961932 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97k4x\" (UniqueName: \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.962590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.961657 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:55.968075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:55.967951 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:56.023348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:56.023314 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97k4x\" (UniqueName: \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\") pod \"maas-api-revoke-sub-8bf4b0d5-zzb8g\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:56.147969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:56.147928 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:56.628032 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:56.627970 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g"] Nov 03 03:49:56.969151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:56.969066 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" event={"ID":"f9a86bb0-922e-4ace-964c-498507c804cd","Type":"ContainerStarted","Data":"360ef8042cc809f3da5d852bfe5286852d3e45701e5161a41830e404437888fb"} Nov 03 03:49:57.976647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:57.976611 5275 generic.go:356] "Generic (PLEG): container finished" podID="f9a86bb0-922e-4ace-964c-498507c804cd" containerID="b058561b3ad8bee8be867c917369dd2efad9709d40f6a80df8d6912333f4466e" exitCode=0 Nov 03 03:49:57.977133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:57.976685 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" event={"ID":"f9a86bb0-922e-4ace-964c-498507c804cd","Type":"ContainerDied","Data":"b058561b3ad8bee8be867c917369dd2efad9709d40f6a80df8d6912333f4466e"} Nov 03 03:49:58.319571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.319527 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds"] Nov 03 03:49:58.323872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.323839 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.338736 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.338704 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsp4b\" (UniqueName: \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.338925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.338825 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.339127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.339109 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.382998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.382965 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds"] Nov 03 03:49:58.440540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.440467 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.440958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.440590 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsp4b\" (UniqueName: \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.440958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.440675 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.441237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.441210 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.443233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.443212 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.501925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.501885 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsp4b\" (UniqueName: \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\") pod \"maas-api-revoke-sub-24d48620-h6nds\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.639109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.639025 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:49:58.969336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:58.969253 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:49:59.026022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.025989 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds"] Nov 03 03:49:59.026526 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:49:59.026372 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879fcb96_30ea_4441_8771_b6a7199ccbba.slice/crio-44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14 WatchSource:0}: Error finding container 44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14: Status 404 returned error can't find the container with id 44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14 Nov 03 03:49:59.075723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.075655 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:49:59.129204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.129177 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:59.147468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.147436 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\") pod \"f9a86bb0-922e-4ace-964c-498507c804cd\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " Nov 03 03:49:59.147633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.147502 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\") pod \"f9a86bb0-922e-4ace-964c-498507c804cd\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " Nov 03 03:49:59.147633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.147559 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\" (UniqueName: \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\") pod \"f9a86bb0-922e-4ace-964c-498507c804cd\" (UID: \"f9a86bb0-922e-4ace-964c-498507c804cd\") " Nov 03 03:49:59.147874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.147832 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca" pod "f9a86bb0-922e-4ace-964c-498507c804cd" (UID: "f9a86bb0-922e-4ace-964c-498507c804cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:49:59.149908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.149881 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x" pod "f9a86bb0-922e-4ace-964c-498507c804cd" (UID: "f9a86bb0-922e-4ace-964c-498507c804cd"). InnerVolumeSpecName "kube-api-access-97k4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:59.150020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.149946 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca" pod "f9a86bb0-922e-4ace-964c-498507c804cd" (UID: "f9a86bb0-922e-4ace-964c-498507c804cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:59.249509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.249414 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:59.249509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.249447 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a86bb0-922e-4ace-964c-498507c804cd-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:59.249509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.249462 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97k4x\" (UniqueName: \"kubernetes.io/projected/f9a86bb0-922e-4ace-964c-498507c804cd-kube-api-access-97k4x\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:59.908644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.908603 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc"] Nov 03 03:49:59.909223 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.909207 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f9a86bb0-922e-4ace-964c-498507c804cd" containerName="revoke-keys" Nov 03 03:49:59.909283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.909224 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a86bb0-922e-4ace-964c-498507c804cd" containerName="revoke-keys" Nov 03 03:49:59.909344 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.909335 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a86bb0-922e-4ace-964c-498507c804cd" containerName="revoke-keys" Nov 03 03:49:59.912774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.912755 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:49:59.959755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.959714 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:49:59.959970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.959921 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:49:59.959970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.959965 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwb9\" (UniqueName: \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:49:59.970375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.970348 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc"] Nov 03 03:49:59.997184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.997153 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" Nov 03 03:49:59.997462 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.997166 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g" event={"ID":"f9a86bb0-922e-4ace-964c-498507c804cd","Type":"ContainerDied","Data":"360ef8042cc809f3da5d852bfe5286852d3e45701e5161a41830e404437888fb"} Nov 03 03:49:59.997620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:49:59.997504 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="360ef8042cc809f3da5d852bfe5286852d3e45701e5161a41830e404437888fb" Nov 03 03:50:00.000127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.000098 5275 generic.go:356] "Generic (PLEG): container finished" podID="879fcb96-30ea-4441-8771-b6a7199ccbba" containerID="f04efd582029a6bb9f80d24f52b822cabf4e2f4d777f328d7ce3990cd9846876" exitCode=0 Nov 03 03:50:00.000272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.000157 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" event={"ID":"879fcb96-30ea-4441-8771-b6a7199ccbba","Type":"ContainerDied","Data":"f04efd582029a6bb9f80d24f52b822cabf4e2f4d777f328d7ce3990cd9846876"} Nov 03 03:50:00.000272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.000199 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" event={"ID":"879fcb96-30ea-4441-8771-b6a7199ccbba","Type":"ContainerStarted","Data":"44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14"} Nov 03 03:50:00.061146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.061103 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.061625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.061174 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwb9\" (UniqueName: \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.061625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.061331 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.061766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.061738 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.063772 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.063750 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.122779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.122743 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwb9\" (UniqueName: \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\") pod \"maas-api-revoke-sub-e30f9fc4-rtwvc\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.310645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.310610 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:00.770750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.770719 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc"] Nov 03 03:50:00.771635 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:00.771603 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad804f21_ab52_4acf_9d5b_f7243146ee66.slice/crio-8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca WatchSource:0}: Error finding container 8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca: Status 404 returned error can't find the container with id 8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca Nov 03 03:50:00.801881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.801836 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:50:00.807448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.807424 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.881610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881565 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.881840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881632 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjx2f\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.881840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881734 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.881840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881770 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.881840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881806 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.882389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.881889 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.882389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.882030 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.882389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.882081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.882389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.882138 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.902436 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.902395 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:50:00.983664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983632 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.983891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983679 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.983891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983720 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.983891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983789 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.983891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983831 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjx2f\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.983891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983880 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983907 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983943 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.983974 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984376 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.984248 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.984442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.984513 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.984581 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.984708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.984663 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.986589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.986566 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:00.986928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:00.986908 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:01.008496 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.008460 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" event={"ID":"ad804f21-ab52-4acf-9d5b-f7243146ee66","Type":"ContainerStarted","Data":"8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca"} Nov 03 03:50:01.046365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.046261 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjx2f\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:01.046365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.046307 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:01.141406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.141381 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:50:01.148278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.148253 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:01.287358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.287325 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\" (UniqueName: \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\") pod \"879fcb96-30ea-4441-8771-b6a7199ccbba\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " Nov 03 03:50:01.287528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.287456 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\") pod \"879fcb96-30ea-4441-8771-b6a7199ccbba\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " Nov 03 03:50:01.287602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.287564 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\") pod \"879fcb96-30ea-4441-8771-b6a7199ccbba\" (UID: \"879fcb96-30ea-4441-8771-b6a7199ccbba\") " Nov 03 03:50:01.288022 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.287901 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca" pod "879fcb96-30ea-4441-8771-b6a7199ccbba" (UID: "879fcb96-30ea-4441-8771-b6a7199ccbba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:01.289484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.289458 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b" pod "879fcb96-30ea-4441-8771-b6a7199ccbba" (UID: "879fcb96-30ea-4441-8771-b6a7199ccbba"). InnerVolumeSpecName "kube-api-access-wsp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:01.289803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.289787 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca" pod "879fcb96-30ea-4441-8771-b6a7199ccbba" (UID: "879fcb96-30ea-4441-8771-b6a7199ccbba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:01.389800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.389709 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsp4b\" (UniqueName: \"kubernetes.io/projected/879fcb96-30ea-4441-8771-b6a7199ccbba-kube-api-access-wsp4b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:01.389800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.389742 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:01.389800 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.389753 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879fcb96-30ea-4441-8771-b6a7199ccbba-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:01.602151 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:01.602117 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4db37c_1ede_4c62_91cb_befeacd0ff5b.slice/crio-8acdca4d653639fe96b51cd48eea1cff596447d6fe47e358026f633655f76086 WatchSource:0}: Error finding container 8acdca4d653639fe96b51cd48eea1cff596447d6fe47e358026f633655f76086: Status 404 returned error can't find the container with id 8acdca4d653639fe96b51cd48eea1cff596447d6fe47e358026f633655f76086 Nov 03 03:50:01.602788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.602722 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:50:01.604810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.604777 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:01.604917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.604883 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:01.604960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:01.604915 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:02.019074 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.018984 5275 generic.go:356] "Generic (PLEG): container finished" podID="ad804f21-ab52-4acf-9d5b-f7243146ee66" containerID="2fa960b74187dea565105f2b1a5bd121a3a194f0da40b2a6de87af44b967b7b6" exitCode=0 Nov 03 03:50:02.019250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.019075 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" event={"ID":"ad804f21-ab52-4acf-9d5b-f7243146ee66","Type":"ContainerDied","Data":"2fa960b74187dea565105f2b1a5bd121a3a194f0da40b2a6de87af44b967b7b6"} Nov 03 03:50:02.020642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.020622 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" Nov 03 03:50:02.020642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.020637 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds" event={"ID":"879fcb96-30ea-4441-8771-b6a7199ccbba","Type":"ContainerDied","Data":"44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14"} Nov 03 03:50:02.020825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.020679 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a1a56c2f2ef2ef722534f3898ed33d00565aed20381cb706beae786fe28c14" Nov 03 03:50:02.022687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.022648 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" event={"ID":"cf4db37c-1ede-4c62-91cb-befeacd0ff5b","Type":"ContainerStarted","Data":"b6a4f09c49c6aa5e4abafc0298ce0840b3a36cee996fb1d47a71448e23d54d38"} Nov 03 03:50:02.022828 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.022695 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" event={"ID":"cf4db37c-1ede-4c62-91cb-befeacd0ff5b","Type":"ContainerStarted","Data":"8acdca4d653639fe96b51cd48eea1cff596447d6fe47e358026f633655f76086"} Nov 03 03:50:02.149021 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.148984 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:02.150653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.150626 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:02.150796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.150676 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:02.276125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:02.276066 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podStartSLOduration=2.276051545 podStartE2EDuration="2.276051545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:02.268454401 +0000 UTC m=+2893.421142823" watchObservedRunningTime="2026-11-03 03:50:02.276051545 +0000 UTC m=+2893.428740039" Nov 03 03:50:02.330490 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:02.330453 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:50:03.179875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.179323 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:03.179875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.179386 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:03.242703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.242668 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:03.313578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.313476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\" (UniqueName: \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\") pod \"ad804f21-ab52-4acf-9d5b-f7243146ee66\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " Nov 03 03:50:03.313578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.313541 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\") pod \"ad804f21-ab52-4acf-9d5b-f7243146ee66\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " Nov 03 03:50:03.313820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.313585 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\") pod \"ad804f21-ab52-4acf-9d5b-f7243146ee66\" (UID: \"ad804f21-ab52-4acf-9d5b-f7243146ee66\") " Nov 03 03:50:03.313972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.313945 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca" pod "ad804f21-ab52-4acf-9d5b-f7243146ee66" (UID: "ad804f21-ab52-4acf-9d5b-f7243146ee66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:03.314144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.314127 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:03.316091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.316067 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca" pod "ad804f21-ab52-4acf-9d5b-f7243146ee66" (UID: "ad804f21-ab52-4acf-9d5b-f7243146ee66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:03.316222 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.316139 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9" pod "ad804f21-ab52-4acf-9d5b-f7243146ee66" (UID: "ad804f21-ab52-4acf-9d5b-f7243146ee66"). InnerVolumeSpecName "kube-api-access-6dwb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:03.415160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.415114 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwb9\" (UniqueName: \"kubernetes.io/projected/ad804f21-ab52-4acf-9d5b-f7243146ee66-kube-api-access-6dwb9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:03.415160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.415143 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad804f21-ab52-4acf-9d5b-f7243146ee66-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:03.970875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:03.969957 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" containerID="cri-o://3877d5aee90fc29b6d3f3172dbe0eedafc250490475a0e4322db75bc6f5032fd" gracePeriod=25 Nov 03 03:50:04.040105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.040072 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" Nov 03 03:50:04.040105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.040090 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc" event={"ID":"ad804f21-ab52-4acf-9d5b-f7243146ee66","Type":"ContainerDied","Data":"8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca"} Nov 03 03:50:04.040402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.040143 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8118d7113a22e8cdb60c97b196dfa393222c8b1366eb53b18528e4cc623544ca" Nov 03 03:50:04.081890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.078106 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" containerID="cri-o://4f5b13a2840a7294338be1a0aba63f3bcf2ebf51f8be541e0c50168806fd9b36" gracePeriod=25 Nov 03 03:50:04.156874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.152316 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:04.156874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.152378 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:04.335207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:04.335165 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:50:05.164874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:05.156291 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:05.164874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:05.156355 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:05.340826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:05.340787 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:50:06.148716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:06.148681 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:06.148945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:06.148735 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:07.165876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:07.164284 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:07.165876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:07.164349 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:08.149029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:08.148986 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:50:08.149250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:08.149061 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:50:09.154158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:09.154125 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:09.154675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:09.154645 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:09.155666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:09.155643 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:50:14.335257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:14.335209 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:50:15.341377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:15.341342 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:50:16.006249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006218 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2"] Nov 03 03:50:16.006789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006775 5275 cpu_manager.go:414] "containerMap: removing container" podUID="879fcb96-30ea-4441-8771-b6a7199ccbba" containerName="revoke-keys" Nov 03 03:50:16.006840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006790 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879fcb96-30ea-4441-8771-b6a7199ccbba" containerName="revoke-keys" Nov 03 03:50:16.006840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006819 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ad804f21-ab52-4acf-9d5b-f7243146ee66" containerName="revoke-keys" Nov 03 03:50:16.006840 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006825 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad804f21-ab52-4acf-9d5b-f7243146ee66" containerName="revoke-keys" Nov 03 03:50:16.006956 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006947 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad804f21-ab52-4acf-9d5b-f7243146ee66" containerName="revoke-keys" Nov 03 03:50:16.006997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.006961 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="879fcb96-30ea-4441-8771-b6a7199ccbba" containerName="revoke-keys" Nov 03 03:50:16.010404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.010382 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.069578 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.069548 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2"] Nov 03 03:50:16.113176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.113119 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:50:16.113176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.113134 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:50:16.113420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.113349 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:50:16.113499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.113452 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:50:16.148692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.148651 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.148922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.148778 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.148922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.148904 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lcm\" (UniqueName: \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.250133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.250092 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lcm\" (UniqueName: \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.250327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.250229 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.250327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.250270 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.251063 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.251040 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.253069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.253045 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.311224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.311180 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lcm\" (UniqueName: \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\") pod \"maas-api-revoke-sub-dc6feda0-lbck2\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.335002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.334974 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:16.718367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:16.718334 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2"] Nov 03 03:50:16.720277 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:16.720249 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf666c9_fac9_4d5b_a55c_253cea8a2b35.slice/crio-c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a WatchSource:0}: Error finding container c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a: Status 404 returned error can't find the container with id c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a Nov 03 03:50:17.171314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:17.171276 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" event={"ID":"8cf666c9-fac9-4d5b-a55c-253cea8a2b35","Type":"ContainerStarted","Data":"aa3851ac038762959a34d76b98ce4055d789938f13455d8f0707c7bb960c1fcc"} Nov 03 03:50:17.171504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:17.171338 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" event={"ID":"8cf666c9-fac9-4d5b-a55c-253cea8a2b35","Type":"ContainerStarted","Data":"c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a"} Nov 03 03:50:17.296666 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:17.296601 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" podStartSLOduration=2.296581012 podStartE2EDuration="2.296581012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:17.290004614 +0000 UTC m=+2908.442693038" watchObservedRunningTime="2026-11-03 03:50:17.296581012 +0000 UTC m=+2908.449269435" Nov 03 03:50:18.199979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:18.199939 5275 generic.go:356] "Generic (PLEG): container finished" podID="8cf666c9-fac9-4d5b-a55c-253cea8a2b35" containerID="aa3851ac038762959a34d76b98ce4055d789938f13455d8f0707c7bb960c1fcc" exitCode=0 Nov 03 03:50:18.200559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:18.200277 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" event={"ID":"8cf666c9-fac9-4d5b-a55c-253cea8a2b35","Type":"ContainerDied","Data":"aa3851ac038762959a34d76b98ce4055d789938f13455d8f0707c7bb960c1fcc"} Nov 03 03:50:19.354442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.354402 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:19.403336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.403302 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\") pod \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " Nov 03 03:50:19.403508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.403409 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\") pod \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " Nov 03 03:50:19.403551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.403507 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\" (UniqueName: \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\") pod \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\" (UID: \"8cf666c9-fac9-4d5b-a55c-253cea8a2b35\") " Nov 03 03:50:19.403869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.403812 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca" pod "8cf666c9-fac9-4d5b-a55c-253cea8a2b35" (UID: "8cf666c9-fac9-4d5b-a55c-253cea8a2b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:19.404112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.404093 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:19.405566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.405536 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca" pod "8cf666c9-fac9-4d5b-a55c-253cea8a2b35" (UID: "8cf666c9-fac9-4d5b-a55c-253cea8a2b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:19.405690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.405669 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm" pod "8cf666c9-fac9-4d5b-a55c-253cea8a2b35" (UID: "8cf666c9-fac9-4d5b-a55c-253cea8a2b35"). InnerVolumeSpecName "kube-api-access-c9lcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:19.505196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.505101 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lcm\" (UniqueName: \"kubernetes.io/projected/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-kube-api-access-c9lcm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:19.505196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.505140 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf666c9-fac9-4d5b-a55c-253cea8a2b35-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:19.979726 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.979684 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765"] Nov 03 03:50:19.980321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.980304 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8cf666c9-fac9-4d5b-a55c-253cea8a2b35" containerName="revoke-keys" Nov 03 03:50:19.980321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.980322 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf666c9-fac9-4d5b-a55c-253cea8a2b35" containerName="revoke-keys" Nov 03 03:50:19.980511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.980439 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf666c9-fac9-4d5b-a55c-253cea8a2b35" containerName="revoke-keys" Nov 03 03:50:19.984115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:19.984093 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.012212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.011508 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.012212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.011616 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.012212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.011794 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmc9x\" (UniqueName: \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.045669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.045628 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765"] Nov 03 03:50:20.112747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.112713 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.112981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.112767 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.112981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.112859 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmc9x\" (UniqueName: \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.113391 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.113369 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.115345 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.115323 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.174389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.174349 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmc9x\" (UniqueName: \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\") pod \"maas-api-revoke-sub-a0dd7d6d-4p765\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.217552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.217520 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" Nov 03 03:50:20.217734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.217559 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2" event={"ID":"8cf666c9-fac9-4d5b-a55c-253cea8a2b35","Type":"ContainerDied","Data":"c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a"} Nov 03 03:50:20.217734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.217619 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87b8b2e96738a89f8302119154a211130dac29bd53b632cadee30a6cd8f6e9a" Nov 03 03:50:20.298590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.298549 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:20.690514 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:20.690478 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258c6669_014c_4403_bf48_e5ad31d35fb8.slice/crio-a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7 WatchSource:0}: Error finding container a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7: Status 404 returned error can't find the container with id a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7 Nov 03 03:50:20.691947 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:20.691917 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765"] Nov 03 03:50:21.227078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:21.226970 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" event={"ID":"258c6669-014c-4403-bf48-e5ad31d35fb8","Type":"ContainerStarted","Data":"fc15bb7580dd1b5b919945e05f5befac3e4270d7bdce2683991bcf660aedbc50"} Nov 03 03:50:21.227078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:21.227032 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" event={"ID":"258c6669-014c-4403-bf48-e5ad31d35fb8","Type":"ContainerStarted","Data":"a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7"} Nov 03 03:50:21.343990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:21.343937 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" podStartSLOduration=2.343923013 podStartE2EDuration="2.343923013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:21.34182108 +0000 UTC m=+2912.494509502" watchObservedRunningTime="2026-11-03 03:50:21.343923013 +0000 UTC m=+2912.496611464" Nov 03 03:50:22.235890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:22.235833 5275 generic.go:356] "Generic (PLEG): container finished" podID="258c6669-014c-4403-bf48-e5ad31d35fb8" containerID="fc15bb7580dd1b5b919945e05f5befac3e4270d7bdce2683991bcf660aedbc50" exitCode=0 Nov 03 03:50:22.236343 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:22.235920 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" event={"ID":"258c6669-014c-4403-bf48-e5ad31d35fb8","Type":"ContainerDied","Data":"fc15bb7580dd1b5b919945e05f5befac3e4270d7bdce2683991bcf660aedbc50"} Nov 03 03:50:22.958133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:22.958094 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:50:22.963425 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:22.963402 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052007 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052154 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052234 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052284 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhztv\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052354 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052440 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052428 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052452 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052570 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.052653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.052635 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.073412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.073372 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:50:23.154163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154114 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154192 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154239 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhztv\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154269 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154370 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154401 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154462 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154559 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.154962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154937 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.155287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154947 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.155287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.154965 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.155287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.155081 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.155542 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.155407 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.156916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.156895 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.157305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.157286 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.215476 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.215401 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.215612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.215511 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhztv\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.279318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.279282 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:23.418465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.418135 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:23.458059 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.458019 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\") pod \"258c6669-014c-4403-bf48-e5ad31d35fb8\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " Nov 03 03:50:23.458233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.458088 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\") pod \"258c6669-014c-4403-bf48-e5ad31d35fb8\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " Nov 03 03:50:23.458233 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.458147 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\" (UniqueName: \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\") pod \"258c6669-014c-4403-bf48-e5ad31d35fb8\" (UID: \"258c6669-014c-4403-bf48-e5ad31d35fb8\") " Nov 03 03:50:23.458444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.458416 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca" pod "258c6669-014c-4403-bf48-e5ad31d35fb8" (UID: "258c6669-014c-4403-bf48-e5ad31d35fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:23.458815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.458793 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:23.460454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.460423 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca" pod "258c6669-014c-4403-bf48-e5ad31d35fb8" (UID: "258c6669-014c-4403-bf48-e5ad31d35fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:23.460564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.460477 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x" pod "258c6669-014c-4403-bf48-e5ad31d35fb8" (UID: "258c6669-014c-4403-bf48-e5ad31d35fb8"). InnerVolumeSpecName "kube-api-access-tmc9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:23.560590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.560556 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258c6669-014c-4403-bf48-e5ad31d35fb8-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:23.560590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.560589 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmc9x\" (UniqueName: \"kubernetes.io/projected/258c6669-014c-4403-bf48-e5ad31d35fb8-kube-api-access-tmc9x\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:23.730295 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:23.730254 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c1bf63_8b06_4b16_b844_74d97525a9c9.slice/crio-8b1c2ac2350acd1be17983ba8c0f02adbc3c30d769544ca5225d2b46fb29f273 WatchSource:0}: Error finding container 8b1c2ac2350acd1be17983ba8c0f02adbc3c30d769544ca5225d2b46fb29f273: Status 404 returned error can't find the container with id 8b1c2ac2350acd1be17983ba8c0f02adbc3c30d769544ca5225d2b46fb29f273 Nov 03 03:50:23.730582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.730557 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:50:23.732696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.732662 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:23.732804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.732758 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:23.732916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:23.732800 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:24.255034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.255002 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" event={"ID":"b5c1bf63-8b06-4b16-b844-74d97525a9c9","Type":"ContainerStarted","Data":"b4ab8da0472af69511a2fe3bbbedba8d5d1e34e659c239ab86a31bdc3b410792"} Nov 03 03:50:24.255196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.255083 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" event={"ID":"b5c1bf63-8b06-4b16-b844-74d97525a9c9","Type":"ContainerStarted","Data":"8b1c2ac2350acd1be17983ba8c0f02adbc3c30d769544ca5225d2b46fb29f273"} Nov 03 03:50:24.257156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.257128 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" event={"ID":"258c6669-014c-4403-bf48-e5ad31d35fb8","Type":"ContainerDied","Data":"a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7"} Nov 03 03:50:24.257272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.257172 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a824cc511e09d208380860c544d5ccfe7290de0be90a6f5390f7082aa56f9cf7" Nov 03 03:50:24.257272 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.257224 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765" Nov 03 03:50:24.261363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.261340 5275 generic.go:356] "Generic (PLEG): container finished" podID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerID="3877d5aee90fc29b6d3f3172dbe0eedafc250490475a0e4322db75bc6f5032fd" exitCode=0 Nov 03 03:50:24.261474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.261407 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" event={"ID":"3c1e5522-7ac0-4368-b248-1d8a0b736147","Type":"ContainerDied","Data":"3877d5aee90fc29b6d3f3172dbe0eedafc250490475a0e4322db75bc6f5032fd"} Nov 03 03:50:24.261474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.261456 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" event={"ID":"3c1e5522-7ac0-4368-b248-1d8a0b736147","Type":"ContainerDied","Data":"1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56"} Nov 03 03:50:24.261549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.261481 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e8904fcc018337ac1fb310e43f39ef570c4a0c391c273571b35e81a3770fb56" Nov 03 03:50:24.263839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.263816 5275 generic.go:356] "Generic (PLEG): container finished" podID="46b823af-7342-4b02-8781-63b5031824f4" containerID="4f5b13a2840a7294338be1a0aba63f3bcf2ebf51f8be541e0c50168806fd9b36" exitCode=0 Nov 03 03:50:24.264045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.264026 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" event={"ID":"46b823af-7342-4b02-8781-63b5031824f4","Type":"ContainerDied","Data":"4f5b13a2840a7294338be1a0aba63f3bcf2ebf51f8be541e0c50168806fd9b36"} Nov 03 03:50:24.282635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.282507 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:24.284683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.284367 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:24.284683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.284420 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:24.290294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.290211 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:50:24.379866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.377590 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\") pod \"3c1e5522-7ac0-4368-b248-1d8a0b736147\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " Nov 03 03:50:24.379866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.377672 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\" (UniqueName: \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\") pod \"3c1e5522-7ac0-4368-b248-1d8a0b736147\" (UID: \"3c1e5522-7ac0-4368-b248-1d8a0b736147\") " Nov 03 03:50:24.389925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.387246 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume" pod "3c1e5522-7ac0-4368-b248-1d8a0b736147" (UID: "3c1e5522-7ac0-4368-b248-1d8a0b736147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:24.389925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.389400 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c1e5522-7ac0-4368-b248-1d8a0b736147-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:24.390245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.390205 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podStartSLOduration=2.390190846 podStartE2EDuration="2.390190846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:24.375573218 +0000 UTC m=+2915.528261640" watchObservedRunningTime="2026-11-03 03:50:24.390190846 +0000 UTC m=+2915.542879266" Nov 03 03:50:24.404367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.404247 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb" pod "3c1e5522-7ac0-4368-b248-1d8a0b736147" (UID: "3c1e5522-7ac0-4368-b248-1d8a0b736147"). InnerVolumeSpecName "kube-api-access-zdznb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:24.446718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.446692 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:50:24.490008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.489963 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\") pod \"46b823af-7342-4b02-8781-63b5031824f4\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " Nov 03 03:50:24.490646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.490244 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\" (UniqueName: \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\") pod \"46b823af-7342-4b02-8781-63b5031824f4\" (UID: \"46b823af-7342-4b02-8781-63b5031824f4\") " Nov 03 03:50:24.490646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.490520 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume" pod "46b823af-7342-4b02-8781-63b5031824f4" (UID: "46b823af-7342-4b02-8781-63b5031824f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:24.491012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.490994 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdznb\" (UniqueName: \"kubernetes.io/projected/3c1e5522-7ac0-4368-b248-1d8a0b736147-kube-api-access-zdznb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:24.491084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.491017 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b823af-7342-4b02-8781-63b5031824f4-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:24.492563 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.492547 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh" pod "46b823af-7342-4b02-8781-63b5031824f4" (UID: "46b823af-7342-4b02-8781-63b5031824f4"). InnerVolumeSpecName "kube-api-access-8dmzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:24.599144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.599108 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dmzh\" (UniqueName: \"kubernetes.io/projected/46b823af-7342-4b02-8781-63b5031824f4-kube-api-access-8dmzh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:24.600750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.600722 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:50:24.600960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:24.600933 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" containerID="cri-o://e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662" gracePeriod=30 Nov 03 03:50:25.279979 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.279947 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:25.280162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.280005 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:25.284211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.284186 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:50:25.284642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.284182 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" event={"ID":"46b823af-7342-4b02-8781-63b5031824f4","Type":"ContainerDied","Data":"7831ac39c37202faba0127c8ae9a56e5c322a254908b132087ab5380b70f5d78"} Nov 03 03:50:25.284642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.284390 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms" Nov 03 03:50:25.284642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.284397 5275 scope.go:122] "RemoveContainer" containerID="4f5b13a2840a7294338be1a0aba63f3bcf2ebf51f8be541e0c50168806fd9b36" Nov 03 03:50:25.334246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.334205 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.157:50052\" within 1s: context deadline exceeded" Nov 03 03:50:25.334444 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.334288 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj" Nov 03 03:50:25.334444 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:25.334308 5275 logging.go:55] [core] [Channel #5864 SubChannel #5865]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.157:50052", ServerName: "10.133.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.157:50052: operation was canceled" Nov 03 03:50:25.477874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.474137 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:50:25.481288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.481251 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-29dd3f-7f5dc65587-hlcms"] Nov 03 03:50:25.577595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.577492 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:50:25.581354 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:25.581327 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-29dd3f-58b76b8bb7-5ppjj"] Nov 03 03:50:26.292891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:26.286359 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:26.292891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:26.286416 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:26.566934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:26.556089 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:50:26.685874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:26.675471 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:50:27.128873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:27.102789 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" path="/var/lib/kubelet/pods/3c1e5522-7ac0-4368-b248-1d8a0b736147/volumes" Nov 03 03:50:27.143873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:27.129911 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b823af-7342-4b02-8781-63b5031824f4" path="/var/lib/kubelet/pods/46b823af-7342-4b02-8781-63b5031824f4/volumes" Nov 03 03:50:27.280513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:27.280478 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:27.280827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:27.280785 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:28.262179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.262133 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:28.292274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.292226 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:28.292604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.292560 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:28.358077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.358018 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:28.358893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.358829 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:50:28.381062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.381003 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:50:28.381431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.381405 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:28.448121 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.439264 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:50:28.979875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.978152 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:50:28.979875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:28.978227 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:50:29.057997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:29.057957 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z"] Nov 03 03:50:29.089861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:29.089798 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a22db0d-x777z"] Nov 03 03:50:29.285871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:29.283121 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:29.285871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:29.283183 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:29.909551 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:29.909522 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:50:30.091716 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.091680 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.092075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092050 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.092285 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092267 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.092487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092470 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.092648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092631 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.092799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092756 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:30.092799 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092768 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.093266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092832 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.093266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092891 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.093266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092948 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:30.093266 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.092959 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\") pod \"7c821073-1948-4520-98e5-ca080b896b1a\" (UID: \"7c821073-1948-4520-98e5-ca080b896b1a\") " Nov 03 03:50:30.093659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.093549 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.093659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.093572 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.094013 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.093988 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:30.094255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.094225 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:30.094457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.094439 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:30.094718 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.094692 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:30.100734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.100695 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:50:30.100931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.100903 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "kube-api-access-dpm8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:30.102405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.102370 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token" pod "7c821073-1948-4520-98e5-ca080b896b1a" (UID: "7c821073-1948-4520-98e5-ca080b896b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194671 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194729 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194740 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c821073-1948-4520-98e5-ca080b896b1a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194748 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194757 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c821073-1948-4520-98e5-ca080b896b1a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194765 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c821073-1948-4520-98e5-ca080b896b1a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.194779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.194775 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpm8l\" (UniqueName: \"kubernetes.io/projected/7c821073-1948-4520-98e5-ca080b896b1a-kube-api-access-dpm8l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:30.280431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.280393 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:30.280616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.280448 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:30.510423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.510343 5275 generic.go:356] "Generic (PLEG): container finished" podID="7c821073-1948-4520-98e5-ca080b896b1a" containerID="e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662" exitCode=0 Nov 03 03:50:30.510423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.510401 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" Nov 03 03:50:30.510909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.510428 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerDied","Data":"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662"} Nov 03 03:50:30.510909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.510489 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk" event={"ID":"7c821073-1948-4520-98e5-ca080b896b1a","Type":"ContainerDied","Data":"63f09b2cb901d39afeea39f9a52088865a0d056b17e36a8e6a79fa1157b4e7ae"} Nov 03 03:50:30.510909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.510514 5275 scope.go:122] "RemoveContainer" containerID="e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662" Nov 03 03:50:30.521140 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.521122 5275 scope.go:122] "RemoveContainer" containerID="e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d" Nov 03 03:50:30.530728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.530711 5275 scope.go:122] "RemoveContainer" containerID="e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662" Nov 03 03:50:30.532150 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:30.532128 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662\": container with ID starting with e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662 not found: ID does not exist" containerID="e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662" Nov 03 03:50:30.532242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.532159 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662"} err="failed to get container status \"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662\": rpc error: code = NotFound desc = could not find container \"e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662\": container with ID starting with e881cf88668d8252c8454c95ef04decbeed31a8a87350508148f5c6c1d139662 not found: ID does not exist" Nov 03 03:50:30.532242 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.532178 5275 scope.go:122] "RemoveContainer" containerID="e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d" Nov 03 03:50:30.532462 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:30.532438 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d\": container with ID starting with e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d not found: ID does not exist" containerID="e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d" Nov 03 03:50:30.532528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.532468 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d"} err="failed to get container status \"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d\": rpc error: code = NotFound desc = could not find container \"e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d\": container with ID starting with e1114131df33e504dabb544306d6bd5f70b86d001ab27b5d06861131365d0a2d not found: ID does not exist" Nov 03 03:50:30.688141 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.688108 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:50:30.692797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.692767 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e84ee2-openshift-default-58879f9b9c-sxlgk"] Nov 03 03:50:30.986760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.986723 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c30a4c-5949-4cb6-9e9e-245f5ddd93f2" path="/var/lib/kubelet/pods/27c30a4c-5949-4cb6-9e9e-245f5ddd93f2/volumes" Nov 03 03:50:30.987327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:30.987307 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c821073-1948-4520-98e5-ca080b896b1a" path="/var/lib/kubelet/pods/7c821073-1948-4520-98e5-ca080b896b1a/volumes" Nov 03 03:50:31.280877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.280822 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 03:50:31.281056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.280910 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 03:50:31.556549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.556452 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" podUID="8e25e042-7773-4ca9-877b-3d6622cbce12" containerName="payload-processing" containerID="cri-o://ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2" gracePeriod=25 Nov 03 03:50:31.676296 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.676252 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" podUID="4897ddb5-3587-4549-b44d-aa7799396dbb" containerName="payload-pre-processing" containerID="cri-o://1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a" gracePeriod=25 Nov 03 03:50:31.854078 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.854040 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:50:31.961094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:31.961057 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:50:32.020869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.018314 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\" (UniqueName: \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\") pod \"8e25e042-7773-4ca9-877b-3d6622cbce12\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " Nov 03 03:50:32.020869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.018384 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\") pod \"8e25e042-7773-4ca9-877b-3d6622cbce12\" (UID: \"8e25e042-7773-4ca9-877b-3d6622cbce12\") " Nov 03 03:50:32.020869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.019630 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume" pod "8e25e042-7773-4ca9-877b-3d6622cbce12" (UID: "8e25e042-7773-4ca9-877b-3d6622cbce12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:32.021504 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.021465 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr" pod "8e25e042-7773-4ca9-877b-3d6622cbce12" (UID: "8e25e042-7773-4ca9-877b-3d6622cbce12"). InnerVolumeSpecName "kube-api-access-q2nzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:32.122415 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.122315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\") pod \"4897ddb5-3587-4549-b44d-aa7799396dbb\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " Nov 03 03:50:32.122605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.122451 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\" (UniqueName: \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\") pod \"4897ddb5-3587-4549-b44d-aa7799396dbb\" (UID: \"4897ddb5-3587-4549-b44d-aa7799396dbb\") " Nov 03 03:50:32.122745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.122723 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nzr\" (UniqueName: \"kubernetes.io/projected/8e25e042-7773-4ca9-877b-3d6622cbce12-kube-api-access-q2nzr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:32.122745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.122740 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e25e042-7773-4ca9-877b-3d6622cbce12-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:32.123651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.123622 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume" pod "4897ddb5-3587-4549-b44d-aa7799396dbb" (UID: "4897ddb5-3587-4549-b44d-aa7799396dbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:32.126351 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.126320 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k" pod "4897ddb5-3587-4549-b44d-aa7799396dbb" (UID: "4897ddb5-3587-4549-b44d-aa7799396dbb"). InnerVolumeSpecName "kube-api-access-lbk4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:32.224325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.224291 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4897ddb5-3587-4549-b44d-aa7799396dbb-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:32.224325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.224320 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbk4k\" (UniqueName: \"kubernetes.io/projected/4897ddb5-3587-4549-b44d-aa7799396dbb-kube-api-access-lbk4k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:32.331269 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:32.331228 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:50:32.530365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.530328 5275 generic.go:356] "Generic (PLEG): container finished" podID="4897ddb5-3587-4549-b44d-aa7799396dbb" containerID="1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a" exitCode=0 Nov 03 03:50:32.530523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.530379 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" Nov 03 03:50:32.530523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.530373 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" event={"ID":"4897ddb5-3587-4549-b44d-aa7799396dbb","Type":"ContainerDied","Data":"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a"} Nov 03 03:50:32.530523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.530508 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx" event={"ID":"4897ddb5-3587-4549-b44d-aa7799396dbb","Type":"ContainerDied","Data":"eb3dcf6a98fbd87c8ddad2b6cf18af3c9fc5969c29fb7144601d6d32909be227"} Nov 03 03:50:32.530523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.530532 5275 scope.go:122] "RemoveContainer" containerID="1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a" Nov 03 03:50:32.532106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.532078 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e25e042-7773-4ca9-877b-3d6622cbce12" containerID="ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2" exitCode=0 Nov 03 03:50:32.532414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.532113 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" Nov 03 03:50:32.532414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.532159 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" event={"ID":"8e25e042-7773-4ca9-877b-3d6622cbce12","Type":"ContainerDied","Data":"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2"} Nov 03 03:50:32.532414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.532198 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv" event={"ID":"8e25e042-7773-4ca9-877b-3d6622cbce12","Type":"ContainerDied","Data":"785222f6f0e33f58a13dd1bbdc4128c42199fc1394f427cd064240eb4a91f58e"} Nov 03 03:50:32.544408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.544381 5275 scope.go:122] "RemoveContainer" containerID="1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a" Nov 03 03:50:32.544752 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:32.544734 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a\": container with ID starting with 1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a not found: ID does not exist" containerID="1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a" Nov 03 03:50:32.544805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.544761 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a"} err="failed to get container status \"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a\": rpc error: code = NotFound desc = could not find container \"1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a\": container with ID starting with 1a7f158c9f7bc3947a21a564cd6b004f39778bdcc3168a16cc28d2dfd043b23a not found: ID does not exist" Nov 03 03:50:32.544805 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.544780 5275 scope.go:122] "RemoveContainer" containerID="ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2" Nov 03 03:50:32.554776 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.554756 5275 scope.go:122] "RemoveContainer" containerID="ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2" Nov 03 03:50:32.555108 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:32.555080 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2\": container with ID starting with ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2 not found: ID does not exist" containerID="ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2" Nov 03 03:50:32.555195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.555113 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2"} err="failed to get container status \"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2\": rpc error: code = NotFound desc = could not find container \"ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2\": container with ID starting with ae51fddd6cce5045af41fa436ec5ac7d966cf8c11e18d82321e9fd83ae7cc7c2 not found: ID does not exist" Nov 03 03:50:32.697084 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.697037 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:50:32.744406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.744371 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ec4ce-78c6cf7fc-bmlcv"] Nov 03 03:50:32.812358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.811936 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:50:32.816892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.816864 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ec4ce-76f48d4767-mzcsx"] Nov 03 03:50:32.989761 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.989715 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4897ddb5-3587-4549-b44d-aa7799396dbb" path="/var/lib/kubelet/pods/4897ddb5-3587-4549-b44d-aa7799396dbb/volumes" Nov 03 03:50:32.990314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:32.990286 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e25e042-7773-4ca9-877b-3d6622cbce12" path="/var/lib/kubelet/pods/8e25e042-7773-4ca9-877b-3d6622cbce12/volumes" Nov 03 03:50:33.274611 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:33.274584 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:33.275192 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:33.275164 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:33.276120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:33.276097 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:50:35.644883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.644833 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c"] Nov 03 03:50:35.645417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645401 5275 cpu_manager.go:414] "containerMap: removing container" podUID="258c6669-014c-4403-bf48-e5ad31d35fb8" containerName="revoke-keys" Nov 03 03:50:35.645465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645418 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c6669-014c-4403-bf48-e5ad31d35fb8" containerName="revoke-keys" Nov 03 03:50:35.645465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645429 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:35.645465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645434 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:35.645465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645448 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e25e042-7773-4ca9-877b-3d6622cbce12" containerName="payload-processing" Nov 03 03:50:35.645465 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645453 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e25e042-7773-4ca9-877b-3d6622cbce12" containerName="payload-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645467 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4897ddb5-3587-4549-b44d-aa7799396dbb" containerName="payload-pre-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645472 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4897ddb5-3587-4549-b44d-aa7799396dbb" containerName="payload-pre-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645482 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645487 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645505 5275 cpu_manager.go:414] "containerMap: removing container" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645510 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645584 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e25e042-7773-4ca9-877b-3d6622cbce12" containerName="payload-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645594 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4897ddb5-3587-4549-b44d-aa7799396dbb" containerName="payload-pre-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645601 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1e5522-7ac0-4368-b248-1d8a0b736147" containerName="payload-processing" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645607 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:35.645617 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645614 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b823af-7342-4b02-8781-63b5031824f4" containerName="payload-pre-processing" Nov 03 03:50:35.645977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645623 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c6669-014c-4403-bf48-e5ad31d35fb8" containerName="revoke-keys" Nov 03 03:50:35.645977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.645629 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:35.650482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.650465 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.665896 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.665863 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnpg\" (UniqueName: \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.666017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.665996 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.666071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.666046 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.707931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.707898 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c"] Nov 03 03:50:35.751980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.751944 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:50:35.751980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.751943 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:50:35.751980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.751979 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:50:35.757655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.757637 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:50:35.767642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.767615 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.767750 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.767663 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.767803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.767775 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnpg\" (UniqueName: \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.768503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.768424 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.770604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.770584 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.831697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.831666 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnpg\" (UniqueName: \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\") pod \"maas-api-revoke-sub-3f622169-mgk8c\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:35.964589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:35.964503 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:36.053196 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.053162 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b"] Nov 03 03:50:36.053714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.053700 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:36.053766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.053715 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c821073-1948-4520-98e5-ca080b896b1a" containerName="istio-proxy" Nov 03 03:50:36.058754 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.058725 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.112857 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.112813 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b"] Nov 03 03:50:36.173901 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.173835 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.174120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.173963 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45z9\" (UniqueName: \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.174181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.174156 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.275804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.275766 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.276030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.275867 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.276030 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.275908 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45z9\" (UniqueName: \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.276616 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.276592 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.278375 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.278355 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.337191 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.337163 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45z9\" (UniqueName: \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\") pod \"maas-api-revoke-sub-6598b2e8-vcd6b\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.346407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.345427 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c"] Nov 03 03:50:36.346407 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:36.345478 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d3f770_18a0_4ff1_a7b0_b943a95715d2.slice/crio-c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512 WatchSource:0}: Error finding container c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512: Status 404 returned error can't find the container with id c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512 Nov 03 03:50:36.373838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.373798 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:36.573970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.573879 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" event={"ID":"a4d3f770-18a0-4ff1-a7b0-b943a95715d2","Type":"ContainerStarted","Data":"c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512"} Nov 03 03:50:36.759613 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:36.759566 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654fd0ae_84eb_402e_b4d8_bb5058710ee8.slice/crio-8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961 WatchSource:0}: Error finding container 8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961: Status 404 returned error can't find the container with id 8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961 Nov 03 03:50:36.760579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:36.760551 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b"] Nov 03 03:50:37.582641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.582535 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" event={"ID":"654fd0ae-84eb-402e-b4d8-bb5058710ee8","Type":"ContainerStarted","Data":"bf02bdcfe2a5c008d81b490c45832337a013fbcd084cfe2387b5d486f4b30dee"} Nov 03 03:50:37.582641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.582601 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" event={"ID":"654fd0ae-84eb-402e-b4d8-bb5058710ee8","Type":"ContainerStarted","Data":"8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961"} Nov 03 03:50:37.584253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.584226 5275 generic.go:356] "Generic (PLEG): container finished" podID="a4d3f770-18a0-4ff1-a7b0-b943a95715d2" containerID="63ab715703ca8d23f96da49802d7af1301f4c94db240118e50a0fb4e69fb6b1b" exitCode=0 Nov 03 03:50:37.584355 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.584283 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" event={"ID":"a4d3f770-18a0-4ff1-a7b0-b943a95715d2","Type":"ContainerDied","Data":"63ab715703ca8d23f96da49802d7af1301f4c94db240118e50a0fb4e69fb6b1b"} Nov 03 03:50:37.699908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.699836 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" podStartSLOduration=1.699822803 podStartE2EDuration="1.699822803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:37.697649749 +0000 UTC m=+2928.850338170" watchObservedRunningTime="2026-11-03 03:50:37.699822803 +0000 UTC m=+2928.852511261" Nov 03 03:50:37.961484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.961402 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:50:37.965628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.965611 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.998789 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.998750 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.998993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.998832 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.998993 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.998908 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999025 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wkf\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999051 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999158 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999107 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999213 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999306 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:37.999364 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:37.999349 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.053031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.052983 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:50:38.100701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100662 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wkf\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.100930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100708 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.100930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100780 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.100930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100816 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.100930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100885 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.100930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100920 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.100956 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101052 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101073 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101169 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101488 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101497 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101589 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101578 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.101787 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.101762 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.103432 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.103409 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.103635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.103618 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.161386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.161351 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.161521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.161415 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wkf\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.279050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.279018 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:38.593315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.593210 5275 generic.go:356] "Generic (PLEG): container finished" podID="654fd0ae-84eb-402e-b4d8-bb5058710ee8" containerID="bf02bdcfe2a5c008d81b490c45832337a013fbcd084cfe2387b5d486f4b30dee" exitCode=0 Nov 03 03:50:38.593315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.593284 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" event={"ID":"654fd0ae-84eb-402e-b4d8-bb5058710ee8","Type":"ContainerDied","Data":"bf02bdcfe2a5c008d81b490c45832337a013fbcd084cfe2387b5d486f4b30dee"} Nov 03 03:50:38.715640 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:38.715604 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8485418b_d237_4fcd_9515_f5a70391fbed.slice/crio-24a42c145966244e9971ef86a7377cb16d045571e0dcf6312b44fd9a9d75b7a1 WatchSource:0}: Error finding container 24a42c145966244e9971ef86a7377cb16d045571e0dcf6312b44fd9a9d75b7a1: Status 404 returned error can't find the container with id 24a42c145966244e9971ef86a7377cb16d045571e0dcf6312b44fd9a9d75b7a1 Nov 03 03:50:38.717871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.717825 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:50:38.718403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.718357 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:38.718530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.718434 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:38.718530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.718468 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:38.724732 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.724710 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:38.810037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.809999 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\") pod \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " Nov 03 03:50:38.810216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.810095 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\") pod \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " Nov 03 03:50:38.810216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.810153 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\" (UniqueName: \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\") pod \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\" (UID: \"a4d3f770-18a0-4ff1-a7b0-b943a95715d2\") " Nov 03 03:50:38.810425 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.810394 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca" pod "a4d3f770-18a0-4ff1-a7b0-b943a95715d2" (UID: "a4d3f770-18a0-4ff1-a7b0-b943a95715d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:38.810679 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.810655 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:38.812511 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.812483 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca" pod "a4d3f770-18a0-4ff1-a7b0-b943a95715d2" (UID: "a4d3f770-18a0-4ff1-a7b0-b943a95715d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:38.813047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.813026 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg" pod "a4d3f770-18a0-4ff1-a7b0-b943a95715d2" (UID: "a4d3f770-18a0-4ff1-a7b0-b943a95715d2"). InnerVolumeSpecName "kube-api-access-fhnpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:38.912681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.912598 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:38.912681 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:38.912643 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnpg\" (UniqueName: \"kubernetes.io/projected/a4d3f770-18a0-4ff1-a7b0-b943a95715d2-kube-api-access-fhnpg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:39.602689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.602652 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" event={"ID":"8485418b-d237-4fcd-9515-f5a70391fbed","Type":"ContainerStarted","Data":"320c7615eec37c3baed6574de1438153f87fc0a467f4a2d14b1a62e575aa59a6"} Nov 03 03:50:39.603184 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.602715 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" event={"ID":"8485418b-d237-4fcd-9515-f5a70391fbed","Type":"ContainerStarted","Data":"24a42c145966244e9971ef86a7377cb16d045571e0dcf6312b44fd9a9d75b7a1"} Nov 03 03:50:39.604195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.604168 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" Nov 03 03:50:39.604314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.604211 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c" event={"ID":"a4d3f770-18a0-4ff1-a7b0-b943a95715d2","Type":"ContainerDied","Data":"c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512"} Nov 03 03:50:39.604314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.604257 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c001e3597a582a094b0b380c8a9f2b267f3ad5e268fb8cf6ea8e8a3a950ec512" Nov 03 03:50:39.729604 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.729542 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podStartSLOduration=2.7295233960000003 podStartE2EDuration="2.729523396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:39.725286522 +0000 UTC m=+2930.877974943" watchObservedRunningTime="2026-11-03 03:50:39.729523396 +0000 UTC m=+2930.882211817" Nov 03 03:50:39.819550 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.819527 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:39.921273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921191 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj"] Nov 03 03:50:39.921766 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921752 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a4d3f770-18a0-4ff1-a7b0-b943a95715d2" containerName="revoke-keys" Nov 03 03:50:39.921817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921767 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d3f770-18a0-4ff1-a7b0-b943a95715d2" containerName="revoke-keys" Nov 03 03:50:39.921817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921792 5275 cpu_manager.go:414] "containerMap: removing container" podUID="654fd0ae-84eb-402e-b4d8-bb5058710ee8" containerName="revoke-keys" Nov 03 03:50:39.921817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921798 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654fd0ae-84eb-402e-b4d8-bb5058710ee8" containerName="revoke-keys" Nov 03 03:50:39.921971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921895 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="654fd0ae-84eb-402e-b4d8-bb5058710ee8" containerName="revoke-keys" Nov 03 03:50:39.921971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.921911 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d3f770-18a0-4ff1-a7b0-b943a95715d2" containerName="revoke-keys" Nov 03 03:50:39.924524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.924504 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\" (UniqueName: \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\") pod \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " Nov 03 03:50:39.924625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.924614 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\") pod \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " Nov 03 03:50:39.924684 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.924671 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\") pod \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\" (UID: \"654fd0ae-84eb-402e-b4d8-bb5058710ee8\") " Nov 03 03:50:39.924981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.924956 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca" pod "654fd0ae-84eb-402e-b4d8-bb5058710ee8" (UID: "654fd0ae-84eb-402e-b4d8-bb5058710ee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:39.925067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.925042 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:39.925484 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.925470 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:39.926755 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.926731 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca" pod "654fd0ae-84eb-402e-b4d8-bb5058710ee8" (UID: "654fd0ae-84eb-402e-b4d8-bb5058710ee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:39.926893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.926825 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9" pod "654fd0ae-84eb-402e-b4d8-bb5058710ee8" (UID: "654fd0ae-84eb-402e-b4d8-bb5058710ee8"). InnerVolumeSpecName "kube-api-access-l45z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:39.999071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:39.999036 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj"] Nov 03 03:50:40.026049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.026013 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.026270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.026058 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.026270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.026086 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.026412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.026372 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd0ae-84eb-402e-b4d8-bb5058710ee8-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:40.026412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.026394 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l45z9\" (UniqueName: \"kubernetes.io/projected/654fd0ae-84eb-402e-b4d8-bb5058710ee8-kube-api-access-l45z9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:40.127039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.127001 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.127039 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.127043 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.127268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.127063 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.127641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.127619 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.129627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.129602 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.190910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.190800 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\") pod \"maas-api-revoke-sub-cac89902-7rwhj\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.283193 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.283160 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:40.287002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.286970 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:40.287268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.287240 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:40.294212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.294185 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:40.628624 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.627624 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" Nov 03 03:50:40.629980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.629947 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b" event={"ID":"654fd0ae-84eb-402e-b4d8-bb5058710ee8","Type":"ContainerDied","Data":"8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961"} Nov 03 03:50:40.630113 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.630029 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8615ce8942838b828b18b1b2859b6d90b222867b2ac881e5f3a9e988248bc961" Nov 03 03:50:40.748127 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:40.748091 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj"] Nov 03 03:50:40.785498 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:40.785461 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ba3767_a5cc_4193_a6cf_78041426c8df.slice/crio-48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d WatchSource:0}: Error finding container 48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d: Status 404 returned error can't find the container with id 48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d Nov 03 03:50:41.281329 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:41.281281 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:41.281539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:41.281342 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:41.635154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:41.635068 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" event={"ID":"96ba3767-a5cc-4193-a6cf-78041426c8df","Type":"ContainerStarted","Data":"c9848d851738c6554ad8a13a255657a55c2bc5af87ad1092b2ccd812dfa2a280"} Nov 03 03:50:41.635154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:41.635121 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" event={"ID":"96ba3767-a5cc-4193-a6cf-78041426c8df","Type":"ContainerStarted","Data":"48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d"} Nov 03 03:50:42.281291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:42.281253 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:42.281483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:42.281389 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:42.644087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:42.644004 5275 generic.go:356] "Generic (PLEG): container finished" podID="96ba3767-a5cc-4193-a6cf-78041426c8df" containerID="c9848d851738c6554ad8a13a255657a55c2bc5af87ad1092b2ccd812dfa2a280" exitCode=0 Nov 03 03:50:42.644455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:42.644180 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" event={"ID":"96ba3767-a5cc-4193-a6cf-78041426c8df","Type":"ContainerDied","Data":"c9848d851738c6554ad8a13a255657a55c2bc5af87ad1092b2ccd812dfa2a280"} Nov 03 03:50:43.282870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:43.282245 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:43.282870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:43.282314 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:43.894204 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:43.894174 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:44.009768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.009683 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\") pod \"96ba3767-a5cc-4193-a6cf-78041426c8df\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " Nov 03 03:50:44.009992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.009814 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\") pod \"96ba3767-a5cc-4193-a6cf-78041426c8df\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " Nov 03 03:50:44.009992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.009958 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\") pod \"96ba3767-a5cc-4193-a6cf-78041426c8df\" (UID: \"96ba3767-a5cc-4193-a6cf-78041426c8df\") " Nov 03 03:50:44.011268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.011218 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca" pod "96ba3767-a5cc-4193-a6cf-78041426c8df" (UID: "96ba3767-a5cc-4193-a6cf-78041426c8df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:44.013775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.013746 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca" pod "96ba3767-a5cc-4193-a6cf-78041426c8df" (UID: "96ba3767-a5cc-4193-a6cf-78041426c8df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:44.013928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.013899 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq" pod "96ba3767-a5cc-4193-a6cf-78041426c8df" (UID: "96ba3767-a5cc-4193-a6cf-78041426c8df"). InnerVolumeSpecName "kube-api-access-trrrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:44.112110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.112079 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:44.112110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.112104 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/96ba3767-a5cc-4193-a6cf-78041426c8df-kube-api-access-trrrq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:44.112110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.112115 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ba3767-a5cc-4193-a6cf-78041426c8df-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:44.280887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.280407 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:44.280887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.280476 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:44.676058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.675964 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" event={"ID":"96ba3767-a5cc-4193-a6cf-78041426c8df","Type":"ContainerDied","Data":"48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d"} Nov 03 03:50:44.676058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.676027 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48abaecba6fd10b7b72a7b8fdf330610e7f48d0481cfd51d7ce40ac964c2577d" Nov 03 03:50:44.676248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:44.676095 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj" Nov 03 03:50:45.280498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:45.280462 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 03:50:45.280935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:45.280521 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 03:50:46.149922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.149881 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:50:46.150892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.150820 5275 cpu_manager.go:414] "containerMap: removing container" podUID="96ba3767-a5cc-4193-a6cf-78041426c8df" containerName="revoke-keys" Nov 03 03:50:46.150892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.150864 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ba3767-a5cc-4193-a6cf-78041426c8df" containerName="revoke-keys" Nov 03 03:50:46.151087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.151030 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ba3767-a5cc-4193-a6cf-78041426c8df" containerName="revoke-keys" Nov 03 03:50:46.155357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.155332 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.212931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.212896 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:50:46.285694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.285661 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:46.286283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.286078 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:46.286970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.286949 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:50:46.342902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.342835 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.342971 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343117 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343038 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343137 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343164 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343184 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343300 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343360 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn948\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.343488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.343474 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.420087 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.420003 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69"] Nov 03 03:50:46.424287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.424263 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.445317 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445285 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445419 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445470 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445537 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445634 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445672 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445700 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445815 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445756 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445811 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn948\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.445946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.445919 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.446205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.446179 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.446322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.446239 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.446431 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.446409 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.447502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.447479 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.448910 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.448841 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.453304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.453276 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.487774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.487242 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:50:46.489124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.489097 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:50:46.489381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.489364 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:50:46.489552 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.489537 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:50:46.495892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.494833 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69"] Nov 03 03:50:46.508784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.508751 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn948\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.509210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.509187 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:46.547139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.547109 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.547330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.547160 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbvk\" (UniqueName: \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.547330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.547307 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.648581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.648525 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbvk\" (UniqueName: \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.648775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.648672 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.648928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.648905 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.649588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.649560 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.651609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.651585 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.710331 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.710264 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbvk\" (UniqueName: \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\") pod \"maas-api-revoke-sub-ad75b3d5-jhk69\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.740613 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.740583 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:46.782920 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:46.782885 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:47.170047 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.170013 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69"] Nov 03 03:50:47.206425 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:47.206387 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a02188c_eb3d_4421_9105_fb1a5dbf46cf.slice/crio-eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10 WatchSource:0}: Error finding container eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10: Status 404 returned error can't find the container with id eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10 Nov 03 03:50:47.222894 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:47.222858 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07864a18_dd3e_479f_b255_933dcf240507.slice/crio-dcde10ddd78281a7e32e2ab0ea34b09d25cceab19ff832a7c5a2907ba933894f WatchSource:0}: Error finding container dcde10ddd78281a7e32e2ab0ea34b09d25cceab19ff832a7c5a2907ba933894f: Status 404 returned error can't find the container with id dcde10ddd78281a7e32e2ab0ea34b09d25cceab19ff832a7c5a2907ba933894f Nov 03 03:50:47.223908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.223865 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:50:47.226034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.226003 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:47.226124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.226066 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:47.226124 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.226095 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:47.707691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.707651 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" event={"ID":"2a02188c-eb3d-4421-9105-fb1a5dbf46cf","Type":"ContainerStarted","Data":"eb547ea750032265f143c5b3045d1ec052e8d91e10bf12672d20864d42cb1cd9"} Nov 03 03:50:47.708163 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.707715 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" event={"ID":"2a02188c-eb3d-4421-9105-fb1a5dbf46cf","Type":"ContainerStarted","Data":"eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10"} Nov 03 03:50:47.710231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.710204 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" event={"ID":"07864a18-dd3e-479f-b255-933dcf240507","Type":"ContainerStarted","Data":"a9d52b77856023bf2535fe9124cbb1f8bd8df71b7b23ed2a299ee86fb21a1a4f"} Nov 03 03:50:47.710358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.710255 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" event={"ID":"07864a18-dd3e-479f-b255-933dcf240507","Type":"ContainerStarted","Data":"dcde10ddd78281a7e32e2ab0ea34b09d25cceab19ff832a7c5a2907ba933894f"} Nov 03 03:50:47.783951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.783912 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:47.826509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:47.826458 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" podStartSLOduration=1.826444037 podStartE2EDuration="1.826444037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:47.82527202 +0000 UTC m=+2938.977960441" watchObservedRunningTime="2026-11-03 03:50:47.826444037 +0000 UTC m=+2938.979132458" Nov 03 03:50:48.745640 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:48.745603 5275 generic.go:356] "Generic (PLEG): container finished" podID="2a02188c-eb3d-4421-9105-fb1a5dbf46cf" containerID="eb547ea750032265f143c5b3045d1ec052e8d91e10bf12672d20864d42cb1cd9" exitCode=0 Nov 03 03:50:48.747307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:48.747277 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" event={"ID":"2a02188c-eb3d-4421-9105-fb1a5dbf46cf","Type":"ContainerDied","Data":"eb547ea750032265f143c5b3045d1ec052e8d91e10bf12672d20864d42cb1cd9"} Nov 03 03:50:48.797948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:48.796921 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:48.797948 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:48.797011 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:48.867005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:48.866940 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podStartSLOduration=2.866919882 podStartE2EDuration="2.866919882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:47.960177274 +0000 UTC m=+2939.112865695" watchObservedRunningTime="2026-11-03 03:50:48.866919882 +0000 UTC m=+2940.019608303" Nov 03 03:50:49.808875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:49.789378 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:49.808875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:49.789457 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:50.343873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.342950 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Nov 03 03:50:50.419626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.399318 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:50.420757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.420718 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:51.034383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.420988 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:50:51.034383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.429033 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:51.034383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.834861 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:51.034383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.032976 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:51.034383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:50.835381 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:51.127873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.127042 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:51.248058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.247946 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:50:51.372342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.372300 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\") pod \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " Nov 03 03:50:51.389055 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.389017 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\") pod \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " Nov 03 03:50:51.401877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.389472 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\" (UniqueName: \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\") pod \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\" (UID: \"2a02188c-eb3d-4421-9105-fb1a5dbf46cf\") " Nov 03 03:50:51.456884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.418409 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:50:51.456884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.427055 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:50:51.456884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.439082 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca" pod "2a02188c-eb3d-4421-9105-fb1a5dbf46cf" (UID: "2a02188c-eb3d-4421-9105-fb1a5dbf46cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:51.456884 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.453824 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:50:51.488881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.467000 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" containerID="cri-o://4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba" gracePeriod=30 Nov 03 03:50:51.505618 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.505502 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:51.551913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.550897 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca" pod "2a02188c-eb3d-4421-9105-fb1a5dbf46cf" (UID: "2a02188c-eb3d-4421-9105-fb1a5dbf46cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:51.551913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.551001 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk" pod "2a02188c-eb3d-4421-9105-fb1a5dbf46cf" (UID: "2a02188c-eb3d-4421-9105-fb1a5dbf46cf"). InnerVolumeSpecName "kube-api-access-xqbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:51.619889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.600039 5275 scope.go:122] "RemoveContainer" containerID="3877d5aee90fc29b6d3f3172dbe0eedafc250490475a0e4322db75bc6f5032fd" Nov 03 03:50:51.619889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.611883 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqbvk\" (UniqueName: \"kubernetes.io/projected/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-kube-api-access-xqbvk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:51.619889 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.611914 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a02188c-eb3d-4421-9105-fb1a5dbf46cf-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:51.783270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.783233 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:51.801904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:51.801825 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:52.007571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.007519 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:52.028891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.008749 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:50:52.028891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.015911 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:50:52.028891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.015994 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:50:52.028891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.016956 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 03 03:50:52.028891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.017569 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:50:52.058400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.036746 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:52.103872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.103328 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:50:52.175994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.160840 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:50:52.235522 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.235490 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:50:52.243026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.242034 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:50:52.245314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.245286 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" event={"ID":"2a02188c-eb3d-4421-9105-fb1a5dbf46cf","Type":"ContainerDied","Data":"eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10"} Nov 03 03:50:52.245446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.245367 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb42c7617b4372df9e46eac0f14b710a521f3e9642fd95e0bc74d91d37341d10" Nov 03 03:50:52.247967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.247942 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69" Nov 03 03:50:52.248677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.248657 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:50:52.248885 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.248839 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 03:50:52.788533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.788496 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:52.788743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:52.788553 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:53.134600 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.134503 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 03:50:53.135379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.135355 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2a02188c-eb3d-4421-9105-fb1a5dbf46cf" containerName="revoke-keys" Nov 03 03:50:53.135493 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.135381 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a02188c-eb3d-4421-9105-fb1a5dbf46cf" containerName="revoke-keys" Nov 03 03:50:53.135546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.135500 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a02188c-eb3d-4421-9105-fb1a5dbf46cf" containerName="revoke-keys" Nov 03 03:50:53.140773 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.140750 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.256873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.247141 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f3e02b-openshift-default-dockercfg-mdx9m\"" Nov 03 03:50:53.285970 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.262648 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5l9\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.326950 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327192 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327242 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327335 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327363 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327402 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327484 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.330874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.327520 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.346519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.341979 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 03:50:53.428553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428463 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428553 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428543 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428619 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428644 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428679 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428784 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.428917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.428826 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.429044 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.429023 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5l9\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.429590 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.429565 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.430053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.430033 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.430187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.430100 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.430256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.430205 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.430363 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.430338 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.431735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.431631 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.432352 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.432334 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.492370 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.492335 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.494645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.493255 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5l9\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\") pod \"e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.510480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.510444 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7"] Nov 03 03:50:53.519455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.519407 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.591374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.591203 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:50:53.591374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.591269 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:50:53.591374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.591289 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:50:53.591374 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.591345 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:50:53.592710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.592680 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7"] Nov 03 03:50:53.633497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.633463 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzz5b\" (UniqueName: \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.633688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.633622 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.633688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.633675 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.735323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.735242 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.735606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.735590 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzz5b\" (UniqueName: \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.735796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.735776 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.736724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.736701 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.737964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.737943 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.767873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.761262 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:53.787325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.787286 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:53.787605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.787579 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:53.798480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.798452 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzz5b\" (UniqueName: \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\") pod \"maas-api-revoke-sub-70b0d1af-d4hg7\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:53.842790 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:53.842757 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:54.088814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.088780 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:50:54.193294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.193257 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:50:54.284026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.280069 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:54.284026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.280166 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:54.284026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.280216 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:50:54.290898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.290834 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 03:50:54.310560 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:50:54.310534 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab433d69_d52a_44ae_b12e_e34b63270c9f.slice/crio-b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f WatchSource:0}: Error finding container b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f: Status 404 returned error can't find the container with id b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f Nov 03 03:50:54.315045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.314818 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7"] Nov 03 03:50:54.396207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.396161 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" event={"ID":"ab433d69-d52a-44ae-b12e-e34b63270c9f","Type":"ContainerStarted","Data":"b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f"} Nov 03 03:50:54.401296 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.401257 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerStarted","Data":"3e10b07d9e856ecf0ca63e2d904fd8aa66fae14db2085e4b94afcc6adb621031"} Nov 03 03:50:54.787335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.787282 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:54.787492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:54.787350 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:55.412074 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.412034 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" event={"ID":"ab433d69-d52a-44ae-b12e-e34b63270c9f","Type":"ContainerStarted","Data":"d19e9bd6cd675f86ca86ff793be7a6725eb9bd422ebd761c5044bb91bfa8b493"} Nov 03 03:50:55.414115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.414077 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerStarted","Data":"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26"} Nov 03 03:50:55.547881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.546907 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" podStartSLOduration=2.54688682 podStartE2EDuration="2.54688682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:55.546129532 +0000 UTC m=+2946.698817957" watchObservedRunningTime="2026-11-03 03:50:55.54688682 +0000 UTC m=+2946.699575243" Nov 03 03:50:55.671327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.671228 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podStartSLOduration=2.671211072 podStartE2EDuration="2.671211072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:55.668063683 +0000 UTC m=+2946.820752103" watchObservedRunningTime="2026-11-03 03:50:55.671211072 +0000 UTC m=+2946.823899492" Nov 03 03:50:55.769035 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.768992 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:55.774723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.774694 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:55.784145 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.784118 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 03 03:50:55.784319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:55.784180 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 03 03:50:56.425781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:56.425742 5275 generic.go:356] "Generic (PLEG): container finished" podID="ab433d69-d52a-44ae-b12e-e34b63270c9f" containerID="d19e9bd6cd675f86ca86ff793be7a6725eb9bd422ebd761c5044bb91bfa8b493" exitCode=0 Nov 03 03:50:56.426303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:56.425780 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" event={"ID":"ab433d69-d52a-44ae-b12e-e34b63270c9f","Type":"ContainerDied","Data":"d19e9bd6cd675f86ca86ff793be7a6725eb9bd422ebd761c5044bb91bfa8b493"} Nov 03 03:50:56.426303 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:56.426270 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:56.427294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:56.427271 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:50:57.015981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.015948 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:50:57.120653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120604 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.120904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.120904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120722 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.120904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120753 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.120904 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120812 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.121155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120918 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.121155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.120961 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.121155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121141 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:57.121324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121232 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:57.121324 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121301 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:57.121442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121332 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.121442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121420 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\") pod \"c323c990-67af-4a0b-9b89-dc5418fc7648\" (UID: \"c323c990-67af-4a0b-9b89-dc5418fc7648\") " Nov 03 03:50:57.121549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121469 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:57.121549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121487 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:57.122037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.121979 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.122037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.122007 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.122037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.122020 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.122037 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.122034 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c323c990-67af-4a0b-9b89-dc5418fc7648-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.122279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.122047 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.124347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.124317 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:50:57.124600 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.124571 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "kube-api-access-n6m2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:57.125004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.124975 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:50:57.125309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.125290 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token" pod "c323c990-67af-4a0b-9b89-dc5418fc7648" (UID: "c323c990-67af-4a0b-9b89-dc5418fc7648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:57.223545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.223502 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6m2f\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-kube-api-access-n6m2f\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.223545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.223544 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.223760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.223559 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.223760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.223576 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c323c990-67af-4a0b-9b89-dc5418fc7648-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.441869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.440552 5275 generic.go:356] "Generic (PLEG): container finished" podID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerID="4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba" exitCode=0 Nov 03 03:50:57.441869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.440950 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerDied","Data":"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba"} Nov 03 03:50:57.441869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.441033 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" event={"ID":"c323c990-67af-4a0b-9b89-dc5418fc7648","Type":"ContainerDied","Data":"b848308f3f2170a12a02dd0a7fa93b771744a3f16db89c1e5bbb3e1b1439847b"} Nov 03 03:50:57.441869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.441052 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9" Nov 03 03:50:57.441869 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.441063 5275 scope.go:122] "RemoveContainer" containerID="4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba" Nov 03 03:50:57.481808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.478039 5275 scope.go:122] "RemoveContainer" containerID="5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b" Nov 03 03:50:57.496150 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.495808 5275 scope.go:122] "RemoveContainer" containerID="4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba" Nov 03 03:50:57.496274 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:57.496222 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba\": container with ID starting with 4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba not found: ID does not exist" containerID="4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba" Nov 03 03:50:57.496358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.496276 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba"} err="failed to get container status \"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba\": rpc error: code = NotFound desc = could not find container \"4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba\": container with ID starting with 4e32f99baf1f1f201bb388babd234ec001837ee4fa4a35c0ae3bac853ee1b0ba not found: ID does not exist" Nov 03 03:50:57.496358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.496296 5275 scope.go:122] "RemoveContainer" containerID="5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b" Nov 03 03:50:57.496688 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:57.496666 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b\": container with ID starting with 5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b not found: ID does not exist" containerID="5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b" Nov 03 03:50:57.496782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.496689 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b"} err="failed to get container status \"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b\": rpc error: code = NotFound desc = could not find container \"5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b\": container with ID starting with 5e6fb8669a5e27b6ba768ce326478cadaf443eee3d98e6b34cf073933f618b2b not found: ID does not exist" Nov 03 03:50:57.637780 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.627337 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:50:57.694609 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.694534 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ec4ce-openshift-default-6f44fd86f7-t86r9"] Nov 03 03:50:57.743089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.743062 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:57.831076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.831031 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:50:57.831248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.831115 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:57.940482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.940452 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\" (UniqueName: \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\") pod \"ab433d69-d52a-44ae-b12e-e34b63270c9f\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " Nov 03 03:50:57.940657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.940494 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\") pod \"ab433d69-d52a-44ae-b12e-e34b63270c9f\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " Nov 03 03:50:57.940745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.940609 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\") pod \"ab433d69-d52a-44ae-b12e-e34b63270c9f\" (UID: \"ab433d69-d52a-44ae-b12e-e34b63270c9f\") " Nov 03 03:50:57.940952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.940921 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca" pod "ab433d69-d52a-44ae-b12e-e34b63270c9f" (UID: "ab433d69-d52a-44ae-b12e-e34b63270c9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:57.941499 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.941408 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:57.943088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.943064 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca" pod "ab433d69-d52a-44ae-b12e-e34b63270c9f" (UID: "ab433d69-d52a-44ae-b12e-e34b63270c9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:57.943173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:57.943098 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b" pod "ab433d69-d52a-44ae-b12e-e34b63270c9f" (UID: "ab433d69-d52a-44ae-b12e-e34b63270c9f"). InnerVolumeSpecName "kube-api-access-pzz5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:58.042810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.042778 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab433d69-d52a-44ae-b12e-e34b63270c9f-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:58.042810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.042808 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzz5b\" (UniqueName: \"kubernetes.io/projected/ab433d69-d52a-44ae-b12e-e34b63270c9f-kube-api-access-pzz5b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:58.365734 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.365707 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:58.366287 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.366268 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:58.366984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.366963 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:50:58.460427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.460394 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" Nov 03 03:50:58.460868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.460400 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7" event={"ID":"ab433d69-d52a-44ae-b12e-e34b63270c9f","Type":"ContainerDied","Data":"b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f"} Nov 03 03:50:58.460868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.460547 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1a1269de2bdeb96f5ebd123699f35fcfdd430df0cbd6108f4f5d76d9926c23f" Nov 03 03:50:58.987861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:58.987813 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" path="/var/lib/kubelet/pods/c323c990-67af-4a0b-9b89-dc5418fc7648/volumes" Nov 03 03:50:59.089605 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.089543 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" podUID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" containerName="payload-processing" containerID="cri-o://8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc" gracePeriod=25 Nov 03 03:50:59.193701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.193662 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" podUID="e286c718-9092-40bd-9f17-7d2a67c52176" containerName="payload-pre-processing" containerID="cri-o://8461dcd33df0ccdd831d1f781ea1c6e1944d0a2663942f4875938e647642e0a2" gracePeriod=25 Nov 03 03:50:59.360295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.360103 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 03:50:59.361086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.360990 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:50:59.361086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361014 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:50:59.361086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361076 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:50:59.361086 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361085 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:50:59.361372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361107 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ab433d69-d52a-44ae-b12e-e34b63270c9f" containerName="revoke-keys" Nov 03 03:50:59.361372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361116 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab433d69-d52a-44ae-b12e-e34b63270c9f" containerName="revoke-keys" Nov 03 03:50:59.361372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361265 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:50:59.361372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.361280 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab433d69-d52a-44ae-b12e-e34b63270c9f" containerName="revoke-keys" Nov 03 03:50:59.365641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.365618 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.386965 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.383653 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:50:59.451902 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.446128 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-10120d-openshift-default-dockercfg-nplqd\"" Nov 03 03:50:59.475342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475002 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\") pod \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " Nov 03 03:50:59.475342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475058 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\" (UniqueName: \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\") pod \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\" (UID: \"40809b23-00a6-4ca7-b7d5-57ecad115ef6\") " Nov 03 03:50:59.475342 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475315 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475359 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475388 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475451 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475478 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475520 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475551 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475668 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tff5\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.475802 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.476601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.476573 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume" pod "40809b23-00a6-4ca7-b7d5-57ecad115ef6" (UID: "40809b23-00a6-4ca7-b7d5-57ecad115ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:59.480019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.479324 5275 generic.go:356] "Generic (PLEG): container finished" podID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" containerID="8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc" exitCode=0 Nov 03 03:50:59.480019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.479544 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" event={"ID":"40809b23-00a6-4ca7-b7d5-57ecad115ef6","Type":"ContainerDied","Data":"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc"} Nov 03 03:50:59.480019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.479595 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" event={"ID":"40809b23-00a6-4ca7-b7d5-57ecad115ef6","Type":"ContainerDied","Data":"8479d93e234cf616c6cd1a4543a7dd312c947d2692b8c6aab2f7da8fcfcf9e3f"} Nov 03 03:50:59.480019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.479623 5275 scope.go:122] "RemoveContainer" containerID="8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc" Nov 03 03:50:59.480019 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.479763 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk" Nov 03 03:50:59.485000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.484812 5275 generic.go:356] "Generic (PLEG): container finished" podID="e286c718-9092-40bd-9f17-7d2a67c52176" containerID="8461dcd33df0ccdd831d1f781ea1c6e1944d0a2663942f4875938e647642e0a2" exitCode=0 Nov 03 03:50:59.485804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.485649 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" event={"ID":"e286c718-9092-40bd-9f17-7d2a67c52176","Type":"ContainerDied","Data":"8461dcd33df0ccdd831d1f781ea1c6e1944d0a2663942f4875938e647642e0a2"} Nov 03 03:50:59.491067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.491045 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 03:50:59.500546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.500509 5275 scope.go:122] "RemoveContainer" containerID="8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc" Nov 03 03:50:59.500990 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:50:59.500963 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc\": container with ID starting with 8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc not found: ID does not exist" containerID="8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc" Nov 03 03:50:59.501173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.501146 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc"} err="failed to get container status \"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc\": rpc error: code = NotFound desc = could not find container \"8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc\": container with ID starting with 8a6255e50b6e3c82fd9d6d540119916f377db86bd72b638bdc881ee0f48e91bc not found: ID does not exist" Nov 03 03:50:59.535051 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.534994 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq" pod "40809b23-00a6-4ca7-b7d5-57ecad115ef6" (UID: "40809b23-00a6-4ca7-b7d5-57ecad115ef6"). InnerVolumeSpecName "kube-api-access-rq7qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:59.577424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577383 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tff5\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577512 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577604 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577637 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577698 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577735 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577724 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577770 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577832 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.577998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577982 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40809b23-00a6-4ca7-b7d5-57ecad115ef6-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:59.578153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.577998 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7qq\" (UniqueName: \"kubernetes.io/projected/40809b23-00a6-4ca7-b7d5-57ecad115ef6-kube-api-access-rq7qq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:59.578531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.578505 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.578827 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.578799 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.579089 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.579070 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.579198 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.579175 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.579308 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.579288 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.581295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.581268 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.582012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.581977 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.604091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.604056 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:50:59.638545 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.638516 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tff5\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.640306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.640279 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\") pod \"e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.679518 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.679471 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\") pod \"e286c718-9092-40bd-9f17-7d2a67c52176\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " Nov 03 03:50:59.679705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.679537 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\" (UniqueName: \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\") pod \"e286c718-9092-40bd-9f17-7d2a67c52176\" (UID: \"e286c718-9092-40bd-9f17-7d2a67c52176\") " Nov 03 03:50:59.679931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.679903 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume" pod "e286c718-9092-40bd-9f17-7d2a67c52176" (UID: "e286c718-9092-40bd-9f17-7d2a67c52176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:59.680155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.680135 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e286c718-9092-40bd-9f17-7d2a67c52176-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:59.681814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.681793 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9" pod "e286c718-9092-40bd-9f17-7d2a67c52176" (UID: "e286c718-9092-40bd-9f17-7d2a67c52176"). InnerVolumeSpecName "kube-api-access-zb2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:59.747704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.747648 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:50:59.781335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.781296 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2l9\" (UniqueName: \"kubernetes.io/projected/e286c718-9092-40bd-9f17-7d2a67c52176-kube-api-access-zb2l9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:59.904712 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.904680 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:50:59.909992 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:50:59.909959 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b4cb2-5d69b9597c-n62qk"] Nov 03 03:51:00.186275 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:00.186247 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbbdb26_48e9_49dd_ad96_4d9ef1ee2211.slice/crio-1a2310e7fcef09a97526df90fdf5247ebec452d3eac28dd806006128f4d8aa5a WatchSource:0}: Error finding container 1a2310e7fcef09a97526df90fdf5247ebec452d3eac28dd806006128f4d8aa5a: Status 404 returned error can't find the container with id 1a2310e7fcef09a97526df90fdf5247ebec452d3eac28dd806006128f4d8aa5a Nov 03 03:51:00.186437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.186317 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 03:51:00.188837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.188801 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:00.188961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.188890 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:00.188961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.188919 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:00.497081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.496985 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerStarted","Data":"3d8b495d1c7d9bb750bc1a40a3cfbcf1a472446c98e8a0c2276715a07b5d04a5"} Nov 03 03:51:00.497081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.497040 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerStarted","Data":"1a2310e7fcef09a97526df90fdf5247ebec452d3eac28dd806006128f4d8aa5a"} Nov 03 03:51:00.498654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.498635 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" Nov 03 03:51:00.498775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.498688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52" event={"ID":"e286c718-9092-40bd-9f17-7d2a67c52176","Type":"ContainerDied","Data":"6fe5ed99bd04ed067d124e53d15352d3722f90505cc8a18c83fab0110b9a6f9a"} Nov 03 03:51:00.498775 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.498741 5275 scope.go:122] "RemoveContainer" containerID="8461dcd33df0ccdd831d1f781ea1c6e1944d0a2663942f4875938e647642e0a2" Nov 03 03:51:00.746873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.742696 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podStartSLOduration=1.742673521 podStartE2EDuration="1.742673521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:00.617660619 +0000 UTC m=+2951.770349041" watchObservedRunningTime="2026-11-03 03:51:00.742673521 +0000 UTC m=+2951.895361943" Nov 03 03:51:00.756962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.756862 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:51:00.765077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.765038 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b4cb2-676f5dd98c-2hn52"] Nov 03 03:51:00.769049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.769019 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:51:00.984573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.984535 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" path="/var/lib/kubelet/pods/40809b23-00a6-4ca7-b7d5-57ecad115ef6/volumes" Nov 03 03:51:00.984997 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:00.984982 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e286c718-9092-40bd-9f17-7d2a67c52176" path="/var/lib/kubelet/pods/e286c718-9092-40bd-9f17-7d2a67c52176/volumes" Nov 03 03:51:01.754210 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:01.754179 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:51:02.332746 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:51:02.332708 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.517085 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.517921 5275 cpu_manager.go:414] "containerMap: removing container" podUID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" containerName="payload-processing" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.517937 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" containerName="payload-processing" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.517959 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e286c718-9092-40bd-9f17-7d2a67c52176" containerName="payload-pre-processing" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.517968 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e286c718-9092-40bd-9f17-7d2a67c52176" containerName="payload-pre-processing" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.518113 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c323c990-67af-4a0b-9b89-dc5418fc7648" containerName="istio-proxy" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.518128 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="40809b23-00a6-4ca7-b7d5-57ecad115ef6" containerName="payload-processing" Nov 03 03:51:02.519062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.518138 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e286c718-9092-40bd-9f17-7d2a67c52176" containerName="payload-pre-processing" Nov 03 03:51:02.539823 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.539055 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.547505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.547475 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:51:02.548779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.548750 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:51:02.585924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.579721 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 03:51:02.617832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.617563 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f3e02b\"" Nov 03 03:51:02.618056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.618023 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f3e02b-dockercfg-jsk6d\"" Nov 03 03:51:02.640747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.640707 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.641090 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.641061 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8sv\" (UniqueName: \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.733825 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.733795 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 03:51:02.737946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.737931 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:02.742882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.742832 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8sv\" (UniqueName: \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.743008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.742961 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.743587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.743566 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.800076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.800048 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 03:51:02.805115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.805087 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8sv\" (UniqueName: \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\") pod \"payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.844561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.844421 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6c4\" (UniqueName: \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:02.847056 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.844664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:02.865582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.865546 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:02.952176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.949948 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:02.952176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.950114 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6c4\" (UniqueName: \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:02.952176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:02.951565 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:03.011723 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.011677 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6c4\" (UniqueName: \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\") pod \"payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:03.051532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.051496 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:03.264834 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.264806 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 03:51:03.446973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.446945 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 03:51:03.447104 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:03.447046 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965f712d_dfe4_4139_9e6e_4012b93ab48b.slice/crio-a082e66310e037586d62efdc37557033444eeb01f94bf2322fb215ca82ccd2de WatchSource:0}: Error finding container a082e66310e037586d62efdc37557033444eeb01f94bf2322fb215ca82ccd2de: Status 404 returned error can't find the container with id a082e66310e037586d62efdc37557033444eeb01f94bf2322fb215ca82ccd2de Nov 03 03:51:03.560379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.560267 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" event={"ID":"22b20891-cb91-487d-a08a-f609f826baee","Type":"ContainerStarted","Data":"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320"} Nov 03 03:51:03.560379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.560337 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" event={"ID":"22b20891-cb91-487d-a08a-f609f826baee","Type":"ContainerStarted","Data":"b589b46d7000051c1d7e7cc8368a396caf9f82b99d825e09c4d7d1d41b77e231"} Nov 03 03:51:03.560634 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.560563 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:03.562261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.562232 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" event={"ID":"965f712d-dfe4-4139-9e6e-4012b93ab48b","Type":"ContainerStarted","Data":"df9a8762180db24809f0bd9f7c56e3a5ce4da524804e91662866707d076ac18e"} Nov 03 03:51:03.562403 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.562281 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" event={"ID":"965f712d-dfe4-4139-9e6e-4012b93ab48b","Type":"ContainerStarted","Data":"a082e66310e037586d62efdc37557033444eeb01f94bf2322fb215ca82ccd2de"} Nov 03 03:51:03.562490 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.562438 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:03.678043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.677991 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" podStartSLOduration=1.6779765389999999 podStartE2EDuration="1.677976539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:03.676941386 +0000 UTC m=+2954.829629807" watchObservedRunningTime="2026-11-03 03:51:03.677976539 +0000 UTC m=+2954.830664960" Nov 03 03:51:03.793911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:03.793837 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" podStartSLOduration=1.793820886 podStartE2EDuration="1.793820886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:03.791619755 +0000 UTC m=+2954.944308177" watchObservedRunningTime="2026-11-03 03:51:03.793820886 +0000 UTC m=+2954.946509328" Nov 03 03:51:07.953528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:07.953492 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:51:07.959275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:07.959250 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.020175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.020142 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:51:08.029700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.029664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.029700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.029712 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.029986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.029835 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.029986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.029906 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.029986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.029944 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnmz\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.030159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.030040 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.030159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.030101 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.030159 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.030138 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.030310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.030179 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.061677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.061428 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\"" Nov 03 03:51:08.131081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131048 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131083 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131116 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131139 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131168 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnmz\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131231 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131270 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131301 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131365 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131341 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131746 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131687 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.131934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.131909 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.132247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.132222 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.132468 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.132449 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.132579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.132561 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.134494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.134465 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.134749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.134725 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.192366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.192315 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.192366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.192346 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnmz\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.275599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.275565 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:08.745938 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:08.745891 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ab3527_cce4_4a60_952a_550e02ad7d34.slice/crio-40c9c162b830adb4c7b6bf659533cec716235d31ba45778c69396b86a6022838 WatchSource:0}: Error finding container 40c9c162b830adb4c7b6bf659533cec716235d31ba45778c69396b86a6022838: Status 404 returned error can't find the container with id 40c9c162b830adb4c7b6bf659533cec716235d31ba45778c69396b86a6022838 Nov 03 03:51:08.746669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.746643 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:51:08.748536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.748503 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:08.748644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.748570 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:08.748644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:08.748599 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:09.613439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.613398 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 03:51:09.619903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.619073 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.624264 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.624206 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 03:51:09.671911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.655181 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" event={"ID":"64ab3527-cce4-4a60-952a-550e02ad7d34","Type":"ContainerStarted","Data":"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2"} Nov 03 03:51:09.671911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.655247 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" event={"ID":"64ab3527-cce4-4a60-952a-550e02ad7d34","Type":"ContainerStarted","Data":"40c9c162b830adb4c7b6bf659533cec716235d31ba45778c69396b86a6022838"} Nov 03 03:51:09.680241 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.680197 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-10120d\"" Nov 03 03:51:09.680492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.680423 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-10120d-dockercfg-pvzt2\"" Nov 03 03:51:09.782938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.782639 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.782938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.782888 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6x6d\" (UniqueName: \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.867927 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.863001 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podStartSLOduration=2.862983169 podStartE2EDuration="2.862983169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:09.862432892 +0000 UTC m=+2961.015121324" watchObservedRunningTime="2026-11-03 03:51:09.862983169 +0000 UTC m=+2961.015671587" Nov 03 03:51:09.883985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.883932 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6x6d\" (UniqueName: \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.884155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.884138 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.884833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.884814 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:09.946510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:09.946473 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6x6d\" (UniqueName: \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\") pod \"payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:10.183838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.183747 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:51:10.239283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.239251 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:10.304875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.300431 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:10.306270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.306235 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:51:10.307980 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.307941 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:10.308109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.307996 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:10.702165 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:10.702126 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b4a33a_3c01_4549_b9b9_a1178b430780.slice/crio-0884febff198a7de22b5fe4c46c02fab7285709e6e41c7fc340cb070bb0f72c1 WatchSource:0}: Error finding container 0884febff198a7de22b5fe4c46c02fab7285709e6e41c7fc340cb070bb0f72c1: Status 404 returned error can't find the container with id 0884febff198a7de22b5fe4c46c02fab7285709e6e41c7fc340cb070bb0f72c1 Nov 03 03:51:10.706430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:10.706398 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 03:51:11.280935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.276311 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:11.280935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.276369 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:11.715626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.715529 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" event={"ID":"58b4a33a-3c01-4549-b9b9-a1178b430780","Type":"ContainerStarted","Data":"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525"} Nov 03 03:51:11.715626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.715594 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" event={"ID":"58b4a33a-3c01-4549-b9b9-a1178b430780","Type":"ContainerStarted","Data":"0884febff198a7de22b5fe4c46c02fab7285709e6e41c7fc340cb070bb0f72c1"} Nov 03 03:51:11.716230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.715902 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:11.843905 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:11.837748 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" podStartSLOduration=2.8377276030000003 podStartE2EDuration="2.837727603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:11.836787045 +0000 UTC m=+2962.989475467" watchObservedRunningTime="2026-11-03 03:51:11.837727603 +0000 UTC m=+2962.990416026" Nov 03 03:51:12.283125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:12.281428 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:12.283125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:12.281486 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:13.284922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:13.283191 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:13.284922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:13.283255 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:14.277880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:14.277285 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:14.277880 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:14.277348 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:14.611213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:14.611135 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 03:51:14.611213 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:14.611203 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 03:51:15.276985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.276941 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 03:51:15.277181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.277022 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 03:51:15.308420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.308356 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" podUID="c6e91555-f47b-4a27-9d64-dba2ec8637df" containerName="payload-pre-processing" containerID="cri-o://e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e" gracePeriod=25 Nov 03 03:51:15.590930 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.590897 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:51:15.691402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.691344 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" podUID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" containerName="payload-processing" containerID="cri-o://fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5" gracePeriod=25 Nov 03 03:51:15.737247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.737211 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\" (UniqueName: \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\") pod \"c6e91555-f47b-4a27-9d64-dba2ec8637df\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " Nov 03 03:51:15.737541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.737380 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\") pod \"c6e91555-f47b-4a27-9d64-dba2ec8637df\" (UID: \"c6e91555-f47b-4a27-9d64-dba2ec8637df\") " Nov 03 03:51:15.737911 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.737885 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume" pod "c6e91555-f47b-4a27-9d64-dba2ec8637df" (UID: "c6e91555-f47b-4a27-9d64-dba2ec8637df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:51:15.740147 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.740118 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p" pod "c6e91555-f47b-4a27-9d64-dba2ec8637df" (UID: "c6e91555-f47b-4a27-9d64-dba2ec8637df"). InnerVolumeSpecName "kube-api-access-bhw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:51:15.764062 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.764024 5275 generic.go:356] "Generic (PLEG): container finished" podID="c6e91555-f47b-4a27-9d64-dba2ec8637df" containerID="e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e" exitCode=0 Nov 03 03:51:15.764249 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.764220 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" event={"ID":"c6e91555-f47b-4a27-9d64-dba2ec8637df","Type":"ContainerDied","Data":"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e"} Nov 03 03:51:15.764319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.764276 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" event={"ID":"c6e91555-f47b-4a27-9d64-dba2ec8637df","Type":"ContainerDied","Data":"fe53df76c1df6e7ca1240c51fe63a6822df20c1b7a2cd82b3d9a56e9a08d6462"} Nov 03 03:51:15.764319 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.764307 5275 scope.go:122] "RemoveContainer" containerID="e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e" Nov 03 03:51:15.765290 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.765256 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d" Nov 03 03:51:15.804512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.804492 5275 scope.go:122] "RemoveContainer" containerID="e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e" Nov 03 03:51:15.804902 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:51:15.804832 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e\": container with ID starting with e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e not found: ID does not exist" containerID="e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e" Nov 03 03:51:15.805008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.804914 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e"} err="failed to get container status \"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e\": rpc error: code = NotFound desc = could not find container \"e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e\": container with ID starting with e2c7f387a836946f5c4d093e7f6fc4c4745238bc063a37b751d5e93aea0c989e not found: ID does not exist" Nov 03 03:51:15.839409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.839330 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw5p\" (UniqueName: \"kubernetes.io/projected/c6e91555-f47b-4a27-9d64-dba2ec8637df-kube-api-access-bhw5p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:15.839409 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.839361 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e91555-f47b-4a27-9d64-dba2ec8637df-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:15.949981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.941376 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:51:15.949981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.948230 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8950af-784858bc-8tv2d"] Nov 03 03:51:15.969449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:15.969425 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:51:16.142946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.142826 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\") pod \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " Nov 03 03:51:16.143122 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.143025 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\" (UniqueName: \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\") pod \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\" (UID: \"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba\") " Nov 03 03:51:16.143318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.143295 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume" pod "6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" (UID: "6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:51:16.143581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.143551 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:16.146034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.146007 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb" pod "6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" (UID: "6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba"). InnerVolumeSpecName "kube-api-access-sx9kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:51:16.245532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.245484 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9kb\" (UniqueName: \"kubernetes.io/projected/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba-kube-api-access-sx9kb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:16.370838 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.370804 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:16.371275 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.371250 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:16.372050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.372026 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:51:16.784316 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.784277 5275 generic.go:356] "Generic (PLEG): container finished" podID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" containerID="fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5" exitCode=0 Nov 03 03:51:16.784801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.784336 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" Nov 03 03:51:16.784801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.784357 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" event={"ID":"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba","Type":"ContainerDied","Data":"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5"} Nov 03 03:51:16.784801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.784421 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x" event={"ID":"6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba","Type":"ContainerDied","Data":"408423cf50a491b3a0916123de792c25cddc9dbc905d7fe57543666ef0542885"} Nov 03 03:51:16.784801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.784453 5275 scope.go:122] "RemoveContainer" containerID="fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5" Nov 03 03:51:16.797579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.797554 5275 scope.go:122] "RemoveContainer" containerID="fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5" Nov 03 03:51:16.797908 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:51:16.797887 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5\": container with ID starting with fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5 not found: ID does not exist" containerID="fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5" Nov 03 03:51:16.797988 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.797916 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5"} err="failed to get container status \"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5\": rpc error: code = NotFound desc = could not find container \"fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5\": container with ID starting with fa1af56cb307595debdf66b20b9b32c8ea4bc4d36d0e113108ebf56e6d2fc5c5 not found: ID does not exist" Nov 03 03:51:16.900683 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.895636 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 03:51:16.901812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.901769 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" containerName="payload-processing" Nov 03 03:51:16.902026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.902009 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" containerName="payload-processing" Nov 03 03:51:16.902151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.902137 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c6e91555-f47b-4a27-9d64-dba2ec8637df" containerName="payload-pre-processing" Nov 03 03:51:16.902234 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.902222 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e91555-f47b-4a27-9d64-dba2ec8637df" containerName="payload-pre-processing" Nov 03 03:51:16.902528 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.902512 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" containerName="payload-processing" Nov 03 03:51:16.902627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.902617 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e91555-f47b-4a27-9d64-dba2ec8637df" containerName="payload-pre-processing" Nov 03 03:51:16.913610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.913584 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:16.919421 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.919253 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj7z\" (UniqueName: \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:16.922878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.919650 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:16.940708 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.940676 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 03:51:16.967664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.967631 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-66d902-dockercfg-pk2k6\"" Nov 03 03:51:16.968508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:16.968484 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-66d902\"" Nov 03 03:51:17.008034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.007998 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e91555-f47b-4a27-9d64-dba2ec8637df" path="/var/lib/kubelet/pods/c6e91555-f47b-4a27-9d64-dba2ec8637df/volumes" Nov 03 03:51:17.008437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.008419 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:51:17.008494 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.008460 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8950af-66d6559fb7-mhw4x"] Nov 03 03:51:17.020245 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.020217 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.020333 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.020262 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj7z\" (UniqueName: \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.020999 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.020980 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.081348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.081251 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj7z\" (UniqueName: \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\") pod \"payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.235384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.235343 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.628837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.628806 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 03:51:17.628980 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:17.628959 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f8aacf_0968_44c1_a39e_db172cba4e3d.slice/crio-ee4b6807b37665d0619180f1d277e2b2fe53ea4b5302044ed76c9853cbbb8df3 WatchSource:0}: Error finding container ee4b6807b37665d0619180f1d277e2b2fe53ea4b5302044ed76c9853cbbb8df3: Status 404 returned error can't find the container with id ee4b6807b37665d0619180f1d277e2b2fe53ea4b5302044ed76c9853cbbb8df3 Nov 03 03:51:17.795043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.795003 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" event={"ID":"36f8aacf-0968-44c1-a39e-db172cba4e3d","Type":"ContainerStarted","Data":"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a"} Nov 03 03:51:17.795464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.795075 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:17.795464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.795101 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" event={"ID":"36f8aacf-0968-44c1-a39e-db172cba4e3d","Type":"ContainerStarted","Data":"ee4b6807b37665d0619180f1d277e2b2fe53ea4b5302044ed76c9853cbbb8df3"} Nov 03 03:51:17.913878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:17.913742 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" podStartSLOduration=1.9137254000000001 podStartE2EDuration="1.9137254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:17.910473538 +0000 UTC m=+2969.063161970" watchObservedRunningTime="2026-11-03 03:51:17.9137254 +0000 UTC m=+2969.066413820" Nov 03 03:51:18.986631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:18.986595 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba" path="/var/lib/kubelet/pods/6dc5b482-4fb4-4a2c-b07e-bf532c5e9aba/volumes" Nov 03 03:51:21.976908 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:21.960388 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 03:51:22.085897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.073273 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.127307 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.126247 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-efa9bf-dockercfg-2rxxw\"" Nov 03 03:51:22.168963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.163839 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.168963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.163989 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4g59\" (UniqueName: \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.205108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.170215 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.205108 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.176037 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-efa9bf\"" Nov 03 03:51:22.242302 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.242197 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.251218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.251173 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 03:51:22.252299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.252257 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.342876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.322432 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 03:51:22.342876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.252706 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 03:51:22.363434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.363371 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.500958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.500035 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4g59\" (UniqueName: \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.500958 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.500286 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.636283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.636243 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.756407 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.756320 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4g59\" (UniqueName: \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\") pod \"payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.787095 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.787053 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 03:51:22.830327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.829084 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:22.938887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:22.937053 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:23.183537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.183493 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 03:51:23.343873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.336224 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 03:51:23.343873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.336562 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 03:51:23.375938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.361555 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:23.375938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.361902 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:23.390871 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.389562 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:23.392614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:23.391495 5275 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 03 03:51:24.177588 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:51:24.171471 5275 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.194s" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.183871 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.183934 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.183982 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.184172 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.184207 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:51:24.190029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.184237 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 03:51:24.202277 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.202241 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 03:51:24.256906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.247133 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 03:51:24.287451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.275404 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 03:51:24.323029 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.313721 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 03:51:24.332389 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.332361 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 03:51:24.938538 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.938501 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" event={"ID":"accbe785-7658-4ba4-b61c-7250e0c1878e","Type":"ContainerStarted","Data":"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f"} Nov 03 03:51:24.938731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.938556 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" event={"ID":"accbe785-7658-4ba4-b61c-7250e0c1878e","Type":"ContainerStarted","Data":"dbee691672fa848c030a325f265a4fed71fb68e375cb75119b3c0e977d649734"} Nov 03 03:51:24.938731 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:24.938720 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:25.418430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:25.418372 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" podStartSLOduration=4.418357363 podStartE2EDuration="4.418357363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:25.414138411 +0000 UTC m=+2976.566826836" watchObservedRunningTime="2026-11-03 03:51:25.418357363 +0000 UTC m=+2976.571045831" Nov 03 03:51:28.806620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:28.806586 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 03:51:32.341360 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:51:32.341327 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:51:35.950913 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:35.950879 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 03:51:37.810173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:37.810141 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:51:37.810622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:37.810359 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" containerID="cri-o://e64f5d8b09090df22908d6033079218f519de2efffab3a151f2ded2990d83cfa" gracePeriod=30 Nov 03 03:51:40.538286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.538248 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:51:40.549598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.549571 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.595641 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.595606 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:51:40.620071 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.620034 5275 patch_prober.go:29] interesting pod/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:51:40.620231 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.620108 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:51:40.639782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.639740 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-158d23-openshift-default-dockercfg-nmv9m\"" Nov 03 03:51:40.647332 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.647308 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:51:40.652299 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.652274 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.675622 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.675860 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676078 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676105 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r888z\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676189 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676291 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676323 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676347 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.689875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.676399 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.704597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.704561 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4d77v\"" Nov 03 03:51:40.707206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.707176 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 03:51:40.707206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.707177 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 03:51:40.707442 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.707247 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 03:51:40.719797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.719682 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:51:40.778135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778098 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.778357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778150 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778174 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778357 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778317 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778380 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778419 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778441 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffc7\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778522 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.778753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778544 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r888z\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.778753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778584 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.778753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778600 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.778753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778668 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.778753 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778747 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779079 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.778776 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779162 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779205 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779260 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779323 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779377 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.779546 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.779462 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.780702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.780677 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.781255 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.781235 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.840176 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.840096 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r888z\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.840751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.840729 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\") pod \"e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.881012 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.880983 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881022 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881074 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881101 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881141 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881178 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffc7\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.881451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.881236 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.882439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.882409 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.883942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.883910 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.884065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.883966 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:40.884181 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.884160 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.884230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.884184 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.884230 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.884170 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.947820 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.947397 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffc7\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.953110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.952353 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fdr5x\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:40.992386 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:40.992359 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:41.339990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.339957 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:51:41.372997 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:41.372963 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a8676c_859c_4956_901e_0408eb4e6e9a.slice/crio-89248b699673acd1682ad9fc877dacf76c44eacf67d465535da383bc7eb7b365 WatchSource:0}: Error finding container 89248b699673acd1682ad9fc877dacf76c44eacf67d465535da383bc7eb7b365: Status 404 returned error can't find the container with id 89248b699673acd1682ad9fc877dacf76c44eacf67d465535da383bc7eb7b365 Nov 03 03:51:41.375874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.375802 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:41.375974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.375938 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:41.376017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.375976 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:41.391177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.391153 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:51:41.393547 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:51:41.393507 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35439b1_b151_439c_ab4e_a6c0f986d059.slice/crio-550e6d149f3fe74f1e2b5dd302d1beb6abd02db7eb65325290e5cc7ac2d8795a WatchSource:0}: Error finding container 550e6d149f3fe74f1e2b5dd302d1beb6abd02db7eb65325290e5cc7ac2d8795a: Status 404 returned error can't find the container with id 550e6d149f3fe74f1e2b5dd302d1beb6abd02db7eb65325290e5cc7ac2d8795a Nov 03 03:51:41.396965 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.396924 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:41.397054 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:41.397035 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:42.091312 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.091272 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerStarted","Data":"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522"} Nov 03 03:51:42.091892 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.091334 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerStarted","Data":"89248b699673acd1682ad9fc877dacf76c44eacf67d465535da383bc7eb7b365"} Nov 03 03:51:42.094918 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.094885 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" event={"ID":"d35439b1-b151-439c-ab4e-a6c0f986d059","Type":"ContainerStarted","Data":"fb1a4c5bf4d0ca51e7b10b325f8a48c9db9bab5ffa326a78a96c40d7a469bddd"} Nov 03 03:51:42.095105 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.094940 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" event={"ID":"d35439b1-b151-439c-ab4e-a6c0f986d059","Type":"ContainerStarted","Data":"550e6d149f3fe74f1e2b5dd302d1beb6abd02db7eb65325290e5cc7ac2d8795a"} Nov 03 03:51:42.095172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.095118 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:42.097269 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.097242 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fdr5x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:51:42.097423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.097296 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:51:42.214283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.214216 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podStartSLOduration=2.214197063 podStartE2EDuration="2.214197063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:42.211870699 +0000 UTC m=+2993.364559116" watchObservedRunningTime="2026-11-03 03:51:42.214197063 +0000 UTC m=+2993.366885486" Nov 03 03:51:42.337870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.336773 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" podStartSLOduration=2.336750566 podStartE2EDuration="2.336750566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:42.334106898 +0000 UTC m=+2993.486795322" watchObservedRunningTime="2026-11-03 03:51:42.336750566 +0000 UTC m=+2993.489438988" Nov 03 03:51:42.884782 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.884737 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:42.893887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:42.893831 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:43.114280 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.114246 5275 generic.go:356] "Generic (PLEG): container finished" podID="ac72609c-50df-45e2-9b87-d5e426c55446" containerID="e64f5d8b09090df22908d6033079218f519de2efffab3a151f2ded2990d83cfa" exitCode=0 Nov 03 03:51:43.115597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.115492 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerDied","Data":"e64f5d8b09090df22908d6033079218f519de2efffab3a151f2ded2990d83cfa"} Nov 03 03:51:43.115597 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.115560 5275 scope.go:122] "RemoveContainer" containerID="d17fec1a99dfddab749d602d7f4a88d22b175135877b41e2f09fc8a111193864" Nov 03 03:51:43.116954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.116928 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:43.117691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.117668 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fdr5x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:51:43.117794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.117717 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:51:43.118739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.118719 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:51:43.141104 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.139789 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:51:43.224566 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224528 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224595 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224684 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224724 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224746 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224787 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224815 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224894 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224916 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\") pod \"ac72609c-50df-45e2-9b87-d5e426c55446\" (UID: \"ac72609c-50df-45e2-9b87-d5e426c55446\") " Nov 03 03:51:43.224987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224964 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:51:43.225240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.224987 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:51:43.225240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.225053 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:51:43.225240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.225154 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:51:43.225240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.225170 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:51:43.226226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.226176 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac72609c-50df-45e2-9b87-d5e426c55446-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.226226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.226207 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.226226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.226221 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.226439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.226236 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.226439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.226251 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.227398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.227365 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:51:43.227527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.227439 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "kube-api-access-kzh5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:51:43.227527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.227463 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:51:43.227797 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.227773 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo" pod "ac72609c-50df-45e2-9b87-d5e426c55446" (UID: "ac72609c-50df-45e2-9b87-d5e426c55446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:51:43.327756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.327720 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.327756 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.327753 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzh5q\" (UniqueName: \"kubernetes.io/projected/ac72609c-50df-45e2-9b87-d5e426c55446-kube-api-access-kzh5q\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.328041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.327764 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac72609c-50df-45e2-9b87-d5e426c55446-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:43.328041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:43.327773 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac72609c-50df-45e2-9b87-d5e426c55446-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:51:44.124985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.124951 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" Nov 03 03:51:44.124985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.124985 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs" event={"ID":"ac72609c-50df-45e2-9b87-d5e426c55446","Type":"ContainerDied","Data":"5b396acfc9f9527ecd3730a59694d6a4d79881c7c20486bde68dccc715494ab2"} Nov 03 03:51:44.125677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.125060 5275 scope.go:122] "RemoveContainer" containerID="e64f5d8b09090df22908d6033079218f519de2efffab3a151f2ded2990d83cfa" Nov 03 03:51:44.406781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.406743 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:51:44.411699 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.411666 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8950af-openshift-default-5bccc598b9-qdcbs"] Nov 03 03:51:44.986757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:44.986723 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" path="/var/lib/kubelet/pods/ac72609c-50df-45e2-9b87-d5e426c55446/volumes" Nov 03 03:51:46.129870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:46.124938 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:51:49.900112 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:49.900039 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:51:49.908427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:49.908401 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:51:52.339197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:52.339167 5275 scope.go:122] "RemoveContainer" containerID="230e2a375a6d347a9254496bbc5138acc36adea0be194b9ddeba50f1648dc97d" Nov 03 03:51:57.077557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.077170 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": read tcp 10.133.0.2:42720->10.133.0.212:15021: read: connection reset by peer" start-of-body= Nov 03 03:51:57.077557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.077259 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": read tcp 10.133.0.2:42720->10.133.0.212:15021: read: connection reset by peer" Nov 03 03:51:57.281722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.281687 5275 generic.go:356] "Generic (PLEG): container finished" podID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerID="de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26" exitCode=0 Nov 03 03:51:57.281934 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.281759 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerDied","Data":"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26"} Nov 03 03:51:57.282188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.282174 5275 scope.go:122] "RemoveContainer" containerID="de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26" Nov 03 03:51:57.283682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.283653 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:57.283757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.283730 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:57.283802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:57.283765 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:51:58.291728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:58.291688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerStarted","Data":"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b"} Nov 03 03:51:58.775898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:58.772245 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:51:58.775898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:58.772399 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 03:51:58.775898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:58.772461 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 03:51:59.774434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:59.774390 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 03:51:59.774939 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:51:59.774465 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 03:52:00.772321 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:00.772284 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 03:52:00.772630 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:00.772593 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 03:52:01.773973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:01.773267 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 03:52:01.773973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:01.773327 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 03:52:02.332087 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:52:02.332052 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:52:02.774928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:02.770257 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 03:52:02.774928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:02.770321 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 03:52:03.772961 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:03.772929 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:52:03.773252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:03.773231 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:52:03.774048 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:03.774026 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 03:52:21.051378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:21.051340 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2"] Nov 03 03:52:21.100701 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:21.100667 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6766f24a-p4tb2"] Nov 03 03:52:22.525745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.525704 5275 generic.go:356] "Generic (PLEG): container finished" podID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerID="3d8b495d1c7d9bb750bc1a40a3cfbcf1a472446c98e8a0c2276715a07b5d04a5" exitCode=0 Nov 03 03:52:22.526248 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.525779 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerDied","Data":"3d8b495d1c7d9bb750bc1a40a3cfbcf1a472446c98e8a0c2276715a07b5d04a5"} Nov 03 03:52:22.526293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.526278 5275 scope.go:122] "RemoveContainer" containerID="3d8b495d1c7d9bb750bc1a40a3cfbcf1a472446c98e8a0c2276715a07b5d04a5" Nov 03 03:52:22.527841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.527813 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:22.527940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.527914 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:22.527984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.527949 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:22.988188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:22.988102 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cf7639-c16e-4add-8737-58ec1b14238b" path="/var/lib/kubelet/pods/91cf7639-c16e-4add-8737-58ec1b14238b/volumes" Nov 03 03:52:23.535898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:23.535861 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerStarted","Data":"6c768a9939a06584319308bfdef82f84bed63d7ab3be494c5f96fa70636b228a"} Nov 03 03:52:23.748673 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:23.748629 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:52:23.748967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:23.748750 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:23.748967 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:23.748792 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:24.748747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:24.748716 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:24.757434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:24.748769 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:25.748941 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:25.748899 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:25.749612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:25.749568 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:26.749053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:26.749015 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:26.749688 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:26.749647 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:27.748334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:27.748293 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:27.748531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:27.748373 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:28.748894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:28.748834 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 03:52:28.749420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:28.748920 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 03:52:29.753402 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:29.753364 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:52:29.753882 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:29.753863 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:52:29.754713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:29.754690 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 03:52:32.337675 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:52:32.337637 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:52:40.063186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:40.063145 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt"] Nov 03 03:52:40.110283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:40.110246 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f2f45-kbjmt"] Nov 03 03:52:40.985238 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:40.985204 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323a16bc-ba7c-4cd4-abee-d987bcb79a05" path="/var/lib/kubelet/pods/323a16bc-ba7c-4cd4-abee-d987bcb79a05/volumes" Nov 03 03:52:55.232016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.231906 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 03:52:55.233620 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.233534 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.233900 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.233882 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.234328 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.234312 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.234437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.234426 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.247175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.247146 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.321876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.318168 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9daccc-dockercfg-d4nxs\"" Nov 03 03:52:55.321876 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.318332 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9daccc\"" Nov 03 03:52:55.396262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.396166 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78r4f\" (UniqueName: \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.396459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.396282 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.396625 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.396596 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 03:52:55.408610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.406283 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 03:52:55.408610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.407409 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.408610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.407426 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac72609c-50df-45e2-9b87-d5e426c55446" containerName="istio-proxy" Nov 03 03:52:55.420714 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.420623 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.447258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.447223 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 03:52:55.497829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.497731 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78r4f\" (UniqueName: \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.497829 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.497787 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.498094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.497905 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.498094 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.497960 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xpz\" (UniqueName: \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.499160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.499129 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.560156 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.560128 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78r4f\" (UniqueName: \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\") pod \"payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.599088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.599050 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.599286 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.599117 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xpz\" (UniqueName: \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.599779 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.599751 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.660450 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.660413 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xpz\" (UniqueName: \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\") pod \"payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:55.698423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.698389 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:55.735569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:55.735538 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:56.054031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.053982 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l"] Nov 03 03:52:56.060585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.060549 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a245853b-c9t5l"] Nov 03 03:52:56.086906 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.086680 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 03:52:56.125885 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:52:56.125835 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57858e7e_b33e_4d64_872b_9ccb4755cda1.slice/crio-da7b0288d0fec2c069ae5e7a4cd9e0212c73ea1c08bc714bd982062363a56377 WatchSource:0}: Error finding container da7b0288d0fec2c069ae5e7a4cd9e0212c73ea1c08bc714bd982062363a56377: Status 404 returned error can't find the container with id da7b0288d0fec2c069ae5e7a4cd9e0212c73ea1c08bc714bd982062363a56377 Nov 03 03:52:56.188515 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.188481 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 03:52:56.189283 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:52:56.189245 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5740fbf4_5b90_46e2_90e8_a7c295dfa4bb.slice/crio-acc43c16a491e4c1474679bda075a3b3351e6ccc2cbdd1c8e8f526ac8797f071 WatchSource:0}: Error finding container acc43c16a491e4c1474679bda075a3b3351e6ccc2cbdd1c8e8f526ac8797f071: Status 404 returned error can't find the container with id acc43c16a491e4c1474679bda075a3b3351e6ccc2cbdd1c8e8f526ac8797f071 Nov 03 03:52:56.928154 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.928114 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" event={"ID":"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb","Type":"ContainerStarted","Data":"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308"} Nov 03 03:52:56.928621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.928194 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:52:56.928621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.928217 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" event={"ID":"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb","Type":"ContainerStarted","Data":"acc43c16a491e4c1474679bda075a3b3351e6ccc2cbdd1c8e8f526ac8797f071"} Nov 03 03:52:56.930120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.930078 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" event={"ID":"57858e7e-b33e-4d64-872b-9ccb4755cda1","Type":"ContainerStarted","Data":"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1"} Nov 03 03:52:56.930247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.930136 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" event={"ID":"57858e7e-b33e-4d64-872b-9ccb4755cda1","Type":"ContainerStarted","Data":"da7b0288d0fec2c069ae5e7a4cd9e0212c73ea1c08bc714bd982062363a56377"} Nov 03 03:52:56.930247 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:56.930165 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:52:57.002074 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:57.002017 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921df2ef-0608-4a9f-a43f-cfebeeaff6c9" path="/var/lib/kubelet/pods/921df2ef-0608-4a9f-a43f-cfebeeaff6c9/volumes" Nov 03 03:52:57.047399 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:57.047342 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" podStartSLOduration=2.047326925 podStartE2EDuration="2.047326925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:57.044786532 +0000 UTC m=+3068.197474953" watchObservedRunningTime="2026-11-03 03:52:57.047326925 +0000 UTC m=+3068.200015345" Nov 03 03:52:57.183696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:57.181596 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" podStartSLOduration=2.181575106 podStartE2EDuration="2.181575106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:57.160434936 +0000 UTC m=+3068.313123359" watchObservedRunningTime="2026-11-03 03:52:57.181575106 +0000 UTC m=+3068.334263528" Nov 03 03:52:58.513653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.513617 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 03:52:58.518079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.518052 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.662868 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.659327 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3d8dc5-openshift-default-dockercfg-hgxj4\"" Nov 03 03:52:58.710516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.710455 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.710722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.710576 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.710722 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.710624 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.710878 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.710763 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpts\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.710987 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.710968 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.711256 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.711228 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.711404 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.711389 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.711527 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.711513 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.711631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.711618 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.715458 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.715428 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 03:52:58.812516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812478 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.812749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812555 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.812749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812596 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.812749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812631 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.812749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812662 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.812749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812726 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812775 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812815 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.812925 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpts\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.813097 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.813235 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.813425 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.813602 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.813428 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.814014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.813648 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.816014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.815985 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.817011 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.816985 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.886075 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.885995 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpts\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:58.888705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:58.888491 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\") pod \"e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:59.149694 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.149584 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:52:59.160076 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.160007 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:52:59.160229 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.160123 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:52:59.631465 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:52:59.631425 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca18c3b7_de5d_44c3_adcb_b6ad1df26a56.slice/crio-16ffd7283fefe4812bd165f16736533bf0ce8c532806d3645a183618a34df6f7 WatchSource:0}: Error finding container 16ffd7283fefe4812bd165f16736533bf0ce8c532806d3645a183618a34df6f7: Status 404 returned error can't find the container with id 16ffd7283fefe4812bd165f16736533bf0ce8c532806d3645a183618a34df6f7 Nov 03 03:52:59.631833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.631462 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 03:52:59.634446 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.634411 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:59.634533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.634476 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:59.634533 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.634507 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:59.961411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.961280 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerStarted","Data":"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029"} Nov 03 03:52:59.961411 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.961355 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerStarted","Data":"16ffd7283fefe4812bd165f16736533bf0ce8c532806d3645a183618a34df6f7"} Nov 03 03:52:59.964135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.964105 5275 generic.go:356] "Generic (PLEG): container finished" podID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerID="b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522" exitCode=0 Nov 03 03:52:59.964294 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.964179 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerDied","Data":"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522"} Nov 03 03:52:59.964705 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.964685 5275 scope.go:122] "RemoveContainer" containerID="b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522" Nov 03 03:52:59.966794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.966749 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:59.966903 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.966873 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:52:59.966951 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:52:59.966925 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:53:00.084111 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.084046 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podStartSLOduration=2.084025573 podStartE2EDuration="2.084025573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:00.082249757 +0000 UTC m=+3071.234938179" watchObservedRunningTime="2026-11-03 03:53:00.084025573 +0000 UTC m=+3071.236713995" Nov 03 03:53:00.149826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.149790 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:53:00.155327 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.155296 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:53:00.988372 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.988338 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:53:00.988897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.988394 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerStarted","Data":"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa"} Nov 03 03:53:00.988897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:00.988467 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:53:01.884837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:01.884802 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:53:01.885058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:01.885013 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 03:53:01.885109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:01.885065 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 03:53:02.330240 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:53:02.330204 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:53:02.884693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:02.884648 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 03:53:02.884933 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:02.884744 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 03:53:03.884568 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:03.884530 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 03:53:03.885027 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:03.884596 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 03:53:04.884441 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:04.884398 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 03:53:04.884632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:04.884468 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 03:53:05.885368 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:05.885330 5275 patch_prober.go:29] interesting pod/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 03:53:05.886002 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:05.885962 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 03:53:06.888648 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:06.888610 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:53:06.889067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:06.888964 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:53:06.889642 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:06.889623 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:53:07.941614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:07.941585 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 03:53:07.942115 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:07.941656 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 03:53:12.145664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.145626 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 03:53:12.150994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.150966 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.204031 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.203986 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-286a8c-openshift-default-dockercfg-jr2dp\"" Nov 03 03:53:12.262480 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.262441 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 03:53:12.331174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331141 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331218 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331429 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331498 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331488 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331527 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331582 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331567 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331638 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331624 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331698 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwnc\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.331751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.331737 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.433830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.433717 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.433830 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.433791 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.433896 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434097 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434134 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434193 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434238 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434268 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434291 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434347 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwnc\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.434974 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.435290 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.435500 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.435368 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.436240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.435558 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.437099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.437071 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.438257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.438231 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.495558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.495519 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.495777 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.495755 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwnc\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\") pod \"e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:12.768760 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:12.768668 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:13.051663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.051615 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4"] Nov 03 03:53:13.055971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.055941 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9382d83e-vmkp4"] Nov 03 03:53:13.214770 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:53:13.214735 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1ea932_483c_42fe_962b_490e4e43f2b6.slice/crio-50f75164beef9435260794b830075730fa62911500dd6b5bae986b5bccf3de45 WatchSource:0}: Error finding container 50f75164beef9435260794b830075730fa62911500dd6b5bae986b5bccf3de45: Status 404 returned error can't find the container with id 50f75164beef9435260794b830075730fa62911500dd6b5bae986b5bccf3de45 Nov 03 03:53:13.215273 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.215251 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 03:53:13.217595 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.217561 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:53:13.217696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.217646 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:53:13.217696 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:13.217675 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:53:14.148467 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.148428 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerStarted","Data":"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f"} Nov 03 03:53:14.148677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.148527 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerStarted","Data":"50f75164beef9435260794b830075730fa62911500dd6b5bae986b5bccf3de45"} Nov 03 03:53:14.351866 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.351789 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podStartSLOduration=2.351775786 podStartE2EDuration="2.351775786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:14.317721929 +0000 UTC m=+3085.470410346" watchObservedRunningTime="2026-11-03 03:53:14.351775786 +0000 UTC m=+3085.504464207" Nov 03 03:53:14.768984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.768945 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:14.775940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.775912 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:14.986145 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:14.986111 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02322c9b-d923-467c-adbc-e65fc2511cf0" path="/var/lib/kubelet/pods/02322c9b-d923-467c-adbc-e65fc2511cf0/volumes" Nov 03 03:53:15.157733 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:15.157693 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:15.158991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:15.158968 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:53:20.050835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:20.050801 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs"] Nov 03 03:53:20.100438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:20.100401 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7038186-xrcjs"] Nov 03 03:53:21.071677 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:21.071638 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6239376e-e219-455f-adc6-f371e71bc6c1" path="/var/lib/kubelet/pods/6239376e-e219-455f-adc6-f371e71bc6c1/volumes" Nov 03 03:53:30.052199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:30.052158 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv"] Nov 03 03:53:30.101226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:30.101190 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4521e-82phv"] Nov 03 03:53:30.988631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:30.988593 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7988fb-b018-4c68-a9aa-f392e92ff16c" path="/var/lib/kubelet/pods/6a7988fb-b018-4c68-a9aa-f392e92ff16c/volumes" Nov 03 03:53:32.052194 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:32.052161 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms"] Nov 03 03:53:32.101530 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:32.101492 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2591e416-qwhms"] Nov 03 03:53:32.331752 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:53:32.331655 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:53:32.985339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:32.985305 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b2bb35-55e9-4380-a73b-fd3f4ce541b3" path="/var/lib/kubelet/pods/f8b2bb35-55e9-4380-a73b-fd3f4ce541b3/volumes" Nov 03 03:53:45.051381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:45.051346 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd"] Nov 03 03:53:45.100771 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:45.100736 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de19551-79gxd"] Nov 03 03:53:46.986394 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:46.986356 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b726bee-ee6c-415c-87c8-2214cf7e8ce3" path="/var/lib/kubelet/pods/9b726bee-ee6c-415c-87c8-2214cf7e8ce3/volumes" Nov 03 03:53:48.050962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:48.050927 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt"] Nov 03 03:53:48.102536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:48.102496 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f688b6-pwkdt"] Nov 03 03:53:48.989081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:48.989045 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4c3fdb-662b-4d08-88d9-379866352e34" path="/var/lib/kubelet/pods/0d4c3fdb-662b-4d08-88d9-379866352e34/volumes" Nov 03 03:53:52.494346 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:52.494312 5275 scope.go:122] "RemoveContainer" containerID="b6b5045ad35a3fbad638711b48efd62ed61aebd84f52a71fe404a95f052ff937" Nov 03 03:53:52.506481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:52.506456 5275 scope.go:122] "RemoveContainer" containerID="ac253e1c32f254a19e68724e3d109e4e665b9cc052a6d516069e66528a5f6275" Nov 03 03:53:59.060774 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:59.060737 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c"] Nov 03 03:53:59.108246 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:53:59.108211 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b9d4b5-skk6c"] Nov 03 03:54:00.988424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:00.988379 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3421e84e-eed0-4638-bc7b-8ed59148999c" path="/var/lib/kubelet/pods/3421e84e-eed0-4638-bc7b-8ed59148999c/volumes" Nov 03 03:54:02.331514 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:54:02.331477 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:54:07.051045 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:07.051005 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn"] Nov 03 03:54:07.100040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:07.100001 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae46d00-mhdmn"] Nov 03 03:54:08.986437 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:08.986402 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0800e6-2d04-4be0-a325-c967caaeec10" path="/var/lib/kubelet/pods/fb0800e6-2d04-4be0-a325-c967caaeec10/volumes" Nov 03 03:54:21.050794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:21.050754 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt"] Nov 03 03:54:21.099938 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:21.099902 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a43c57e-qb8pt"] Nov 03 03:54:22.988632 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:22.988551 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ad702d-04f1-4887-98fe-e3db19178a95" path="/var/lib/kubelet/pods/33ad702d-04f1-4887-98fe-e3db19178a95/volumes" Nov 03 03:54:25.050817 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:25.050780 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf"] Nov 03 03:54:25.100423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:25.100376 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e385438-m7lgf"] Nov 03 03:54:27.022870 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:27.022204 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c791978-64cf-4803-8dac-300897e4d032" path="/var/lib/kubelet/pods/7c791978-64cf-4803-8dac-300897e4d032/volumes" Nov 03 03:54:32.330814 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:54:32.330779 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:54:37.942974 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.942941 5275 generic.go:356] "Generic (PLEG): container finished" podID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerID="7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029" exitCode=0 Nov 03 03:54:37.943430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.943015 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerDied","Data":"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029"} Nov 03 03:54:37.943497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.943450 5275 scope.go:122] "RemoveContainer" containerID="7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029" Nov 03 03:54:37.945068 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.945037 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:54:37.945172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.945104 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:54:37.945172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:37.945137 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:54:38.992428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:38.992394 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerStarted","Data":"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d"} Nov 03 03:54:39.165293 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:39.165260 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:54:39.167909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:39.165599 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:39.168172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:39.168141 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:40.158874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:40.153221 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:40.158874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:40.153295 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:41.154886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:41.154334 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:41.154886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:41.154431 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:42.170875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:42.162315 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:42.170875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:42.162375 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:43.156875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:43.155293 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:43.156875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:43.155363 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:44.166874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:44.159304 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 03:54:44.166874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:44.159370 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 03:54:45.050603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:45.050564 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d"] Nov 03 03:54:45.099977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:45.099943 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aca150-6tf5d"] Nov 03 03:54:45.155388 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:45.155355 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:54:45.155717 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:45.155681 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:54:45.156481 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:45.156463 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 03:54:46.985240 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:46.985199 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b409d49f-4957-4fdf-b77c-e5589fbf0882" path="/var/lib/kubelet/pods/b409d49f-4957-4fdf-b77c-e5589fbf0882/volumes" Nov 03 03:54:52.549587 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.549556 5275 scope.go:122] "RemoveContainer" containerID="c80f8cff525cb02d529c8800111e15eb6c30ce9677d6b5322936004e1e1a9bf1" Nov 03 03:54:52.563474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.563440 5275 scope.go:122] "RemoveContainer" containerID="f32e2f11a68d416dc868b8004ad6658267a05c859b1aecc062d312581d050379" Nov 03 03:54:52.576573 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.576546 5275 scope.go:122] "RemoveContainer" containerID="5515fa668af79259d62637b7dafd7ade5c7e6cda2cf60de4ea6f1bc9ba4e07f4" Nov 03 03:54:52.587395 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.587374 5275 scope.go:122] "RemoveContainer" containerID="544ea8c33a8e6efa3bbf8fd61d766d22e6146de10590e6e13c625181358a8d67" Nov 03 03:54:52.600384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.600358 5275 scope.go:122] "RemoveContainer" containerID="353bcd2e7a55911a5f8cf801fd42e3ccd598aca9d92d712ec8bcda6ad422f884" Nov 03 03:54:52.610621 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.610600 5275 scope.go:122] "RemoveContainer" containerID="40af18ca561faab2c98350b8ff1165e6abc342ed433e8e545dcde04c25ed3c49" Nov 03 03:54:52.621523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:54:52.621502 5275 scope.go:122] "RemoveContainer" containerID="073d17fb2145c9fc5fb08ab04f288be758b76ae3f4afbc899111ed8db1602456" Nov 03 03:55:00.087924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:00.086491 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g"] Nov 03 03:55:00.105748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:00.105714 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf4b0d5-zzb8g"] Nov 03 03:55:00.986651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:00.986608 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a86bb0-922e-4ace-964c-498507c804cd" path="/var/lib/kubelet/pods/f9a86bb0-922e-4ace-964c-498507c804cd/volumes" Nov 03 03:55:02.054887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:02.053765 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds"] Nov 03 03:55:02.101320 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:02.101279 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d48620-h6nds"] Nov 03 03:55:02.332045 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:55:02.331951 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:55:02.990334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:02.990297 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879fcb96-30ea-4441-8771-b6a7199ccbba" path="/var/lib/kubelet/pods/879fcb96-30ea-4441-8771-b6a7199ccbba/volumes" Nov 03 03:55:04.074881 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:04.072936 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc"] Nov 03 03:55:04.099675 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:04.099638 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f9fc4-rtwvc"] Nov 03 03:55:04.992921 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:04.992887 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad804f21-ab52-4acf-9d5b-f7243146ee66" path="/var/lib/kubelet/pods/ad804f21-ab52-4acf-9d5b-f7243146ee66/volumes" Nov 03 03:55:09.253966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.253882 5275 generic.go:356] "Generic (PLEG): container finished" podID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerID="9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f" exitCode=0 Nov 03 03:55:09.253966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.253929 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerDied","Data":"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f"} Nov 03 03:55:09.254451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.254371 5275 scope.go:122] "RemoveContainer" containerID="9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f" Nov 03 03:55:09.256175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.256144 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:55:09.256281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.256217 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:55:09.256281 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:09.256249 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:55:10.279914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:10.275397 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerStarted","Data":"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501"} Nov 03 03:55:10.774079 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:10.774045 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:55:10.775877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:10.775813 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 03 03:55:10.776151 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:10.776116 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 03 03:55:11.774877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:11.773323 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 03 03:55:11.774877 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:11.773405 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 03 03:55:12.771872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:12.771327 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 03 03:55:12.771872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:12.771408 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 03 03:55:13.772330 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:13.772288 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 03 03:55:13.772983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:13.772948 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 03 03:55:14.770483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:14.770446 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 03 03:55:14.770778 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:14.770751 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 03 03:55:15.160875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:15.159944 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:55:15.774651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:15.774609 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:55:15.775964 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:15.775939 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 03:55:20.056898 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:20.056866 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2"] Nov 03 03:55:20.106118 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:20.106080 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6feda0-lbck2"] Nov 03 03:55:20.985646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:20.985610 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf666c9-fac9-4d5b-a55c-253cea8a2b35" path="/var/lib/kubelet/pods/8cf666c9-fac9-4d5b-a55c-253cea8a2b35/volumes" Nov 03 03:55:24.049693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:24.049658 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765"] Nov 03 03:55:24.100177 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:24.100140 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0dd7d6d-4p765"] Nov 03 03:55:24.988026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:24.987982 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258c6669-014c-4403-bf48-e5ad31d35fb8" path="/var/lib/kubelet/pods/258c6669-014c-4403-bf48-e5ad31d35fb8/volumes" Nov 03 03:55:32.334162 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:55:32.334122 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:55:39.060612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:39.060568 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c"] Nov 03 03:55:39.108981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:39.108942 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f622169-mgk8c"] Nov 03 03:55:40.051662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:40.051622 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b"] Nov 03 03:55:40.100835 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:40.100794 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6598b2e8-vcd6b"] Nov 03 03:55:40.987644 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:40.987607 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654fd0ae-84eb-402e-b4d8-bb5058710ee8" path="/var/lib/kubelet/pods/654fd0ae-84eb-402e-b4d8-bb5058710ee8/volumes" Nov 03 03:55:40.988126 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:40.988109 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d3f770-18a0-4ff1-a7b0-b943a95715d2" path="/var/lib/kubelet/pods/a4d3f770-18a0-4ff1-a7b0-b943a95715d2/volumes" Nov 03 03:55:44.085295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:44.085266 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj"] Nov 03 03:55:44.134387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:44.134355 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac89902-7rwhj"] Nov 03 03:55:44.987560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:44.987516 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ba3767-a5cc-4193-a6cf-78041426c8df" path="/var/lib/kubelet/pods/96ba3767-a5cc-4193-a6cf-78041426c8df/volumes" Nov 03 03:55:52.054768 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.054727 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69"] Nov 03 03:55:52.104448 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.104415 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75b3d5-jhk69"] Nov 03 03:55:52.575592 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.575552 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w"] Nov 03 03:55:52.579808 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.579789 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.682991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.682959 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:55:52.682991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.682960 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:55:52.683227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.683016 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:55:52.683227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.683033 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 03:55:52.686443 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.686418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w"] Nov 03 03:55:52.692664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.692633 5275 scope.go:122] "RemoveContainer" containerID="acb65241bdbbe72f0a08033fced0c15554f088efe8a903906ccc7047bd1dc7b2" Nov 03 03:55:52.704274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.704253 5275 scope.go:122] "RemoveContainer" containerID="8e8be5b849751317c91e83f8fb590b7873808f8b542797192f10294e7a734f13" Nov 03 03:55:52.705322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.705304 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.705385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.705354 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.705608 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.705590 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.715066 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.715038 5275 scope.go:122] "RemoveContainer" containerID="2a81bdb386c653c52eae995d62fbc305205fa4a681d7585dd6efc2c2391f81a5" Nov 03 03:55:52.732172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.732135 5275 scope.go:122] "RemoveContainer" containerID="8477318d3fdacca6ef4ae15a407c96c31dfeb0308d8896bd0cb70f39e46f37e7" Nov 03 03:55:52.742464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.742439 5275 scope.go:122] "RemoveContainer" containerID="6c4061bd27838ef7e0c4614b0eb5515be4b11581c3682c3db31d374cc66d4fde" Nov 03 03:55:52.806580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.806541 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.806781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.806610 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.806781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.806650 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.807387 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.807360 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.809221 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.809197 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.868007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.867937 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\") pod \"maas-api-revoke-keys-e2e-explicit-158d23-j8s2w\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.893917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.893882 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:52.989919 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:52.989882 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a02188c-eb3d-4421-9105-fb1a5dbf46cf" path="/var/lib/kubelet/pods/2a02188c-eb3d-4421-9105-fb1a5dbf46cf/volumes" Nov 03 03:55:53.278202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:53.278172 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w"] Nov 03 03:55:53.279178 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:55:53.279138 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989a28ca_7911_44f3_a658_35a98513088a.slice/crio-96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695 WatchSource:0}: Error finding container 96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695: Status 404 returned error can't find the container with id 96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695 Nov 03 03:55:53.687704 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:53.687660 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" event={"ID":"989a28ca-7911-44f3-a658-35a98513088a","Type":"ContainerStarted","Data":"6f783bb344398ec9d59754d755cc1ad8c47c292ce99426d1787607e7c02ce3cc"} Nov 03 03:55:53.687960 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:53.687724 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" event={"ID":"989a28ca-7911-44f3-a658-35a98513088a","Type":"ContainerStarted","Data":"96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695"} Nov 03 03:55:53.804763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:53.804707 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" podStartSLOduration=1.8046880079999998 podStartE2EDuration="1.804688008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:55:53.803227236 +0000 UTC m=+3244.955915658" watchObservedRunningTime="2026-11-03 03:55:53.804688008 +0000 UTC m=+3244.957376429" Nov 03 03:55:54.700610 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:54.700519 5275 generic.go:356] "Generic (PLEG): container finished" podID="989a28ca-7911-44f3-a658-35a98513088a" containerID="6f783bb344398ec9d59754d755cc1ad8c47c292ce99426d1787607e7c02ce3cc" exitCode=0 Nov 03 03:55:54.701040 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:54.700620 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" event={"ID":"989a28ca-7911-44f3-a658-35a98513088a","Type":"ContainerDied","Data":"6f783bb344398ec9d59754d755cc1ad8c47c292ce99426d1787607e7c02ce3cc"} Nov 03 03:55:55.893335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:55.893305 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:56.055003 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.054968 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\") pod \"989a28ca-7911-44f3-a658-35a98513088a\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " Nov 03 03:55:56.055250 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.055231 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\") pod \"989a28ca-7911-44f3-a658-35a98513088a\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " Nov 03 03:55:56.055335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.055293 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\") pod \"989a28ca-7911-44f3-a658-35a98513088a\" (UID: \"989a28ca-7911-44f3-a658-35a98513088a\") " Nov 03 03:55:56.055691 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.055658 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca" pod "989a28ca-7911-44f3-a658-35a98513088a" (UID: "989a28ca-7911-44f3-a658-35a98513088a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:55:56.056636 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.056436 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989a28ca-7911-44f3-a658-35a98513088a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:55:56.057526 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.057452 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca" pod "989a28ca-7911-44f3-a658-35a98513088a" (UID: "989a28ca-7911-44f3-a658-35a98513088a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:55:56.058046 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.058023 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8" pod "989a28ca-7911-44f3-a658-35a98513088a" (UID: "989a28ca-7911-44f3-a658-35a98513088a"). InnerVolumeSpecName "kube-api-access-7xtc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:55:56.158459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.158417 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989a28ca-7911-44f3-a658-35a98513088a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:55:56.158459 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.158458 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/989a28ca-7911-44f3-a658-35a98513088a-kube-api-access-7xtc8\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:55:56.717393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.717359 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" Nov 03 03:55:56.717579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.717386 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w" event={"ID":"989a28ca-7911-44f3-a658-35a98513088a","Type":"ContainerDied","Data":"96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695"} Nov 03 03:55:56.717579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:56.717462 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96dce4ba2c7c01c250df9b17ac0e19f14a08c82f4350f5d109804b9318a99695" Nov 03 03:55:58.050929 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:58.050894 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7"] Nov 03 03:55:58.100720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:58.100682 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b0d1af-d4hg7"] Nov 03 03:55:58.986558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:55:58.986524 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab433d69-d52a-44ae-b12e-e34b63270c9f" path="/var/lib/kubelet/pods/ab433d69-d52a-44ae-b12e-e34b63270c9f/volumes" Nov 03 03:56:02.333546 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:56:02.333507 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:56:32.332918 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:56:32.332876 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:56:49.958405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:49.958371 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:56:49.967810 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:49.967772 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 03:56:52.794931 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.794895 5275 scope.go:122] "RemoveContainer" containerID="eb547ea750032265f143c5b3045d1ec052e8d91e10bf12672d20864d42cb1cd9" Nov 03 03:56:52.805807 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.805785 5275 scope.go:122] "RemoveContainer" containerID="bf02bdcfe2a5c008d81b490c45832337a013fbcd084cfe2387b5d486f4b30dee" Nov 03 03:56:52.816083 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.816057 5275 scope.go:122] "RemoveContainer" containerID="b058561b3ad8bee8be867c917369dd2efad9709d40f6a80df8d6912333f4466e" Nov 03 03:56:52.826439 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.826408 5275 scope.go:122] "RemoveContainer" containerID="c9848d851738c6554ad8a13a255657a55c2bc5af87ad1092b2ccd812dfa2a280" Nov 03 03:56:52.840796 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.840766 5275 scope.go:122] "RemoveContainer" containerID="fc15bb7580dd1b5b919945e05f5befac3e4270d7bdce2683991bcf660aedbc50" Nov 03 03:56:52.851531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.851510 5275 scope.go:122] "RemoveContainer" containerID="63ab715703ca8d23f96da49802d7af1301f4c94db240118e50a0fb4e69fb6b1b" Nov 03 03:56:52.861298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.861273 5275 scope.go:122] "RemoveContainer" containerID="f04efd582029a6bb9f80d24f52b822cabf4e2f4d777f328d7ce3990cd9846876" Nov 03 03:56:52.870598 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.870575 5275 scope.go:122] "RemoveContainer" containerID="2fa960b74187dea565105f2b1a5bd121a3a194f0da40b2a6de87af44b967b7b6" Nov 03 03:56:52.880318 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:52.880293 5275 scope.go:122] "RemoveContainer" containerID="aa3851ac038762959a34d76b98ce4055d789938f13455d8f0707c7bb960c1fcc" Nov 03 03:56:53.009932 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.009899 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:56:53.010168 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.010117 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" containerID="cri-o://b4ab8da0472af69511a2fe3bbbedba8d5d1e34e659c239ab86a31bdc3b410792" gracePeriod=30 Nov 03 03:56:53.059923 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.059813 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:56:53.060132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.060094 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" containerID="cri-o://b0494fe516652a7b10755c8173e9072cf4532a8568c5943fedbb69e6d8864a4a" gracePeriod=30 Nov 03 03:56:53.063460 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.063434 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:56:53.063669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.063644 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" containerID="cri-o://320c7615eec37c3baed6574de1438153f87fc0a467f4a2d14b1a62e575aa59a6" gracePeriod=30 Nov 03 03:56:53.111698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.111662 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:56:53.111983 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:53.111937 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" containerID="cri-o://f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf" gracePeriod=30 Nov 03 03:56:58.207165 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.207125 5275 generic.go:356] "Generic (PLEG): container finished" podID="8485418b-d237-4fcd-9515-f5a70391fbed" containerID="320c7615eec37c3baed6574de1438153f87fc0a467f4a2d14b1a62e575aa59a6" exitCode=0 Nov 03 03:56:58.207667 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.207298 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" event={"ID":"8485418b-d237-4fcd-9515-f5a70391fbed","Type":"ContainerDied","Data":"320c7615eec37c3baed6574de1438153f87fc0a467f4a2d14b1a62e575aa59a6"} Nov 03 03:56:58.209341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.209311 5275 generic.go:356] "Generic (PLEG): container finished" podID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerID="b4ab8da0472af69511a2fe3bbbedba8d5d1e34e659c239ab86a31bdc3b410792" exitCode=0 Nov 03 03:56:58.209471 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.209386 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" event={"ID":"b5c1bf63-8b06-4b16-b844-74d97525a9c9","Type":"ContainerDied","Data":"b4ab8da0472af69511a2fe3bbbedba8d5d1e34e659c239ab86a31bdc3b410792"} Nov 03 03:56:58.210890 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.210872 5275 generic.go:356] "Generic (PLEG): container finished" podID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerID="b0494fe516652a7b10755c8173e9072cf4532a8568c5943fedbb69e6d8864a4a" exitCode=0 Nov 03 03:56:58.210991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.210943 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" event={"ID":"e5104b7b-3513-4c05-9afe-a77545cc7b9f","Type":"ContainerDied","Data":"b0494fe516652a7b10755c8173e9072cf4532a8568c5943fedbb69e6d8864a4a"} Nov 03 03:56:58.369954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.369922 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:56:58.454916 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.454883 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:56:58.458412 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.458388 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.464982 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465026 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465062 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465106 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465137 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465242 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465272 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465341 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465375 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\") pod \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\" (UID: \"b5c1bf63-8b06-4b16-b844-74d97525a9c9\") " Nov 03 03:56:58.465752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.465526 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.466428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.466107 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.466428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.466254 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.466739 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.466706 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:58.467066 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.467046 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.467179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.467071 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.467179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.467087 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.467179 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.467101 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.468631 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.468606 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.469100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.469066 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:58.469218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.469191 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.470466 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.470442 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "kube-api-access-qhztv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.483875 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.483790 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy" pod "b5c1bf63-8b06-4b16-b844-74d97525a9c9" (UID: "b5c1bf63-8b06-4b16-b844-74d97525a9c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.497430 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.497400 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:56:58.567737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.567704 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.567966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.567819 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.567966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.567886 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.567966 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.567952 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568010 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.568146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568066 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568146 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568098 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568153 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568199 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568258 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568287 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568302 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568328 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568373 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568449 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568479 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568525 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568510 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568579 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\") pod \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\" (UID: \"e5104b7b-3513-4c05-9afe-a77545cc7b9f\") " Nov 03 03:56:58.568981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568611 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568648 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\") pod \"8485418b-d237-4fcd-9515-f5a70391fbed\" (UID: \"8485418b-d237-4fcd-9515-f5a70391fbed\") " Nov 03 03:56:58.568981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568750 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.568981 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.568959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.569261 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569172 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.569325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569287 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhztv\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-kube-api-access-qhztv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569310 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569324 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569339 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569342 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569353 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569367 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8485418b-d237-4fcd-9515-f5a70391fbed-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569380 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569394 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569407 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569455 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569419 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569433 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5c1bf63-8b06-4b16-b844-74d97525a9c9-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569478 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.569962 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.569914 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.570132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.570069 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.571432 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "kube-api-access-rdrdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.571499 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.572993 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "kube-api-access-l4wkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.573223 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.573192 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.573343 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy" pod "e5104b7b-3513-4c05-9afe-a77545cc7b9f" (UID: "e5104b7b-3513-4c05-9afe-a77545cc7b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.573561 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.576887 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.575162 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy" pod "8485418b-d237-4fcd-9515-f5a70391fbed" (UID: "8485418b-d237-4fcd-9515-f5a70391fbed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.670752 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.670669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.670955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.670779 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.670955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.670810 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.670955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.670881 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.670955 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.670949 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671007 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671051 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671093 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671156 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671187 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.671212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671211 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\") pod \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\" (UID: \"c2d5cdee-ba5f-4169-ac86-4a10114549fd\") " Nov 03 03:56:58.671541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671206 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:58.671541 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671316 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.671826 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671802 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671829 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671888 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671904 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdrdf\" (UniqueName: \"kubernetes.io/projected/e5104b7b-3513-4c05-9afe-a77545cc7b9f-kube-api-access-rdrdf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671913 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671919 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671961 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wkf\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-kube-api-access-l4wkf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671977 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8485418b-d237-4fcd-9515-f5a70391fbed-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.671998 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672011 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672020 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8485418b-d237-4fcd-9515-f5a70391fbed-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672028 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8485418b-d237-4fcd-9515-f5a70391fbed-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672037 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672045 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672057 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.672129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.672071 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5104b7b-3513-4c05-9afe-a77545cc7b9f-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.673749 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.673726 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.673922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.673901 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "kube-api-access-nqztj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.673982 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.673965 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:58.674023 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.673985 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy" pod "c2d5cdee-ba5f-4169-ac86-4a10114549fd" (UID: "c2d5cdee-ba5f-4169-ac86-4a10114549fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.775007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.774792 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.775007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.774831 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d5cdee-ba5f-4169-ac86-4a10114549fd-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.775007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.774891 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqztj\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-kube-api-access-nqztj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.775007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.774908 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.775007 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:58.774922 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d5cdee-ba5f-4169-ac86-4a10114549fd-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:59.223160 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.223126 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" Nov 03 03:56:59.223670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.223132 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn" event={"ID":"8485418b-d237-4fcd-9515-f5a70391fbed","Type":"ContainerDied","Data":"24a42c145966244e9971ef86a7377cb16d045571e0dcf6312b44fd9a9d75b7a1"} Nov 03 03:56:59.223670 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.223298 5275 scope.go:122] "RemoveContainer" containerID="320c7615eec37c3baed6574de1438153f87fc0a467f4a2d14b1a62e575aa59a6" Nov 03 03:56:59.224837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.224805 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" event={"ID":"b5c1bf63-8b06-4b16-b844-74d97525a9c9","Type":"ContainerDied","Data":"8b1c2ac2350acd1be17983ba8c0f02adbc3c30d769544ca5225d2b46fb29f273"} Nov 03 03:56:59.224837 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.224819 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5" Nov 03 03:56:59.226519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.226494 5275 generic.go:356] "Generic (PLEG): container finished" podID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerID="f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf" exitCode=0 Nov 03 03:56:59.226659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.226557 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" Nov 03 03:56:59.226659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.226575 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" event={"ID":"c2d5cdee-ba5f-4169-ac86-4a10114549fd","Type":"ContainerDied","Data":"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf"} Nov 03 03:56:59.226659 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.226626 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94" event={"ID":"c2d5cdee-ba5f-4169-ac86-4a10114549fd","Type":"ContainerDied","Data":"0bb8b6d0411655167437e2e4ac5e49b22ace6aa8ac583922fb7af3d75c1c5ae4"} Nov 03 03:56:59.228601 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.228582 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" Nov 03 03:56:59.228751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.228611 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq" event={"ID":"e5104b7b-3513-4c05-9afe-a77545cc7b9f","Type":"ContainerDied","Data":"db4ea4f6ad9d3a71e98a0f4865740d2a36f4890a7cd87d8ad4dcc10cf8718189"} Nov 03 03:56:59.235646 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.235628 5275 scope.go:122] "RemoveContainer" containerID="b4ab8da0472af69511a2fe3bbbedba8d5d1e34e659c239ab86a31bdc3b410792" Nov 03 03:56:59.245584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.245564 5275 scope.go:122] "RemoveContainer" containerID="f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf" Nov 03 03:56:59.259397 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.259370 5275 scope.go:122] "RemoveContainer" containerID="f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf" Nov 03 03:56:59.259757 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:56:59.259733 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf\": container with ID starting with f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf not found: ID does not exist" containerID="f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf" Nov 03 03:56:59.259886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.259767 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf"} err="failed to get container status \"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf\": rpc error: code = NotFound desc = could not find container \"f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf\": container with ID starting with f5fe6a2c8c1ee72f03b96287628db9a1a1c1e3af5b22f0cbfd0c35dd6b6317cf not found: ID does not exist" Nov 03 03:56:59.259886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.259793 5275 scope.go:122] "RemoveContainer" containerID="b0494fe516652a7b10755c8173e9072cf4532a8568c5943fedbb69e6d8864a4a" Nov 03 03:56:59.396719 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.396684 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:56:59.445689 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.445657 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-44bd5"] Nov 03 03:56:59.514004 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.513921 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:56:59.515174 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.515153 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-4z6nq"] Nov 03 03:56:59.626725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.626688 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:56:59.637942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.637909 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-8dvqn"] Nov 03 03:56:59.747649 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.747614 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:56:59.754423 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:56:59.754392 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-bqq94"] Nov 03 03:57:00.987512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:00.987474 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" path="/var/lib/kubelet/pods/8485418b-d237-4fcd-9515-f5a70391fbed/volumes" Nov 03 03:57:00.988227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:00.988192 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" path="/var/lib/kubelet/pods/b5c1bf63-8b06-4b16-b844-74d97525a9c9/volumes" Nov 03 03:57:00.988788 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:00.988771 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" path="/var/lib/kubelet/pods/c2d5cdee-ba5f-4169-ac86-4a10114549fd/volumes" Nov 03 03:57:00.989861 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:00.989816 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" path="/var/lib/kubelet/pods/e5104b7b-3513-4c05-9afe-a77545cc7b9f/volumes" Nov 03 03:57:02.331488 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:57:02.331454 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:57:04.028008 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.027964 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:57:04.028651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.028245 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" containerID="cri-o://5c4cf5db2051f8886ffdadcf1b31dc45e12a98f810247177a98125a8fce6ad89" gracePeriod=30 Nov 03 03:57:04.077968 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.077937 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:57:04.078211 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.078150 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" containerID="cri-o://a9d52b77856023bf2535fe9124cbb1f8bd8df71b7b23ed2a299ee86fb21a1a4f" gracePeriod=30 Nov 03 03:57:04.080710 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.080672 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:57:04.080914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.080888 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" containerID="cri-o://e87cf69df2e35202a6adcf7d6b435b33d95c417b15cf7af7cdea6d170e0ba9f9" gracePeriod=30 Nov 03 03:57:04.127557 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.127519 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:57:04.127759 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:04.127714 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" containerID="cri-o://b6a4f09c49c6aa5e4abafc0298ce0840b3a36cee996fb1d47a71448e23d54d38" gracePeriod=30 Nov 03 03:57:08.136053 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:08.136017 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:57:08.136487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:08.136254 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" containerID="cri-o://1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2" gracePeriod=30 Nov 03 03:57:09.062314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.062276 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:57:09.062492 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.062335 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:57:09.156175 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.156142 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 03:57:09.156560 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.156208 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 03:57:09.375627 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.375572 5275 generic.go:356] "Generic (PLEG): container finished" podID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerID="b6a4f09c49c6aa5e4abafc0298ce0840b3a36cee996fb1d47a71448e23d54d38" exitCode=0 Nov 03 03:57:09.376000 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.375713 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" event={"ID":"cf4db37c-1ede-4c62-91cb-befeacd0ff5b","Type":"ContainerDied","Data":"b6a4f09c49c6aa5e4abafc0298ce0840b3a36cee996fb1d47a71448e23d54d38"} Nov 03 03:57:09.391189 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.391153 5275 generic.go:356] "Generic (PLEG): container finished" podID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerID="5c4cf5db2051f8886ffdadcf1b31dc45e12a98f810247177a98125a8fce6ad89" exitCode=0 Nov 03 03:57:09.391379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.391279 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerDied","Data":"5c4cf5db2051f8886ffdadcf1b31dc45e12a98f810247177a98125a8fce6ad89"} Nov 03 03:57:09.391379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.391338 5275 scope.go:122] "RemoveContainer" containerID="52abbf6dfc05e17e71c48a48f49caf88ebe2c3df56285793cdf1fccb5904ed23" Nov 03 03:57:09.398069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.397803 5275 generic.go:356] "Generic (PLEG): container finished" podID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerID="e87cf69df2e35202a6adcf7d6b435b33d95c417b15cf7af7cdea6d170e0ba9f9" exitCode=0 Nov 03 03:57:09.398069 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.397983 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" event={"ID":"02c8d3bc-75c6-46a0-bfd5-c458128904de","Type":"ContainerDied","Data":"e87cf69df2e35202a6adcf7d6b435b33d95c417b15cf7af7cdea6d170e0ba9f9"} Nov 03 03:57:09.400622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.400376 5275 generic.go:356] "Generic (PLEG): container finished" podID="07864a18-dd3e-479f-b255-933dcf240507" containerID="a9d52b77856023bf2535fe9124cbb1f8bd8df71b7b23ed2a299ee86fb21a1a4f" exitCode=0 Nov 03 03:57:09.400622 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.400548 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" event={"ID":"07864a18-dd3e-479f-b255-933dcf240507","Type":"ContainerDied","Data":"a9d52b77856023bf2535fe9124cbb1f8bd8df71b7b23ed2a299ee86fb21a1a4f"} Nov 03 03:57:09.520400 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.520363 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:57:09.577335 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.577263 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:57:09.644713 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.644688 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:57:09.648226 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.648203 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:57:09.652635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652610 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652704 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652735 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652768 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652797 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652826 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652900 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.652990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652919 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.652999 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653021 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653237 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653244 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653276 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653319 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.653417 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653411 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653484 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653532 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653539 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653548 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653565 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653623 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\") pod \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\" (UID: \"a12a6b3d-1e9d-403f-9108-519c4b4bc898\") " Nov 03 03:57:09.653765 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653703 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\") pod \"07864a18-dd3e-479f-b255-933dcf240507\" (UID: \"07864a18-dd3e-479f-b255-933dcf240507\") " Nov 03 03:57:09.654199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653883 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.654199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.653979 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.654199 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654147 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.654514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654491 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:09.654594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654541 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654568 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654594 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654583 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654598 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654611 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654624 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07864a18-dd3e-479f-b255-933dcf240507-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654636 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654655 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.654791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.654668 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.656107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.656076 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "kube-api-access-bn948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.656207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.656185 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.656509 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.656489 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "kube-api-access-zkcfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.656897 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.656826 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.657070 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.657046 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:09.657803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.657789 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:09.658093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.658076 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token" pod "07864a18-dd3e-479f-b255-933dcf240507" (UID: "07864a18-dd3e-479f-b255-933dcf240507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.658197 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.658177 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy" pod "a12a6b3d-1e9d-403f-9108-519c4b4bc898" (UID: "a12a6b3d-1e9d-403f-9108-519c4b4bc898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.755100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755058 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755100 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755103 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755139 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755163 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755223 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755296 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755329 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755360 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755416 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.755454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755432 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755479 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755520 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755562 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755643 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755684 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755724 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755757 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755802 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\") pod \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\" (UID: \"cf4db37c-1ede-4c62-91cb-befeacd0ff5b\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755877 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756132 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.755936 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\") pod \"02c8d3bc-75c6-46a0-bfd5-c458128904de\" (UID: \"02c8d3bc-75c6-46a0-bfd5-c458128904de\") " Nov 03 03:57:09.756813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756786 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.756813 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756797 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756828 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756862 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756879 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07864a18-dd3e-479f-b255-933dcf240507-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756894 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcfp\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-kube-api-access-zkcfp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756909 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07864a18-dd3e-479f-b255-933dcf240507-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756923 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756937 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn948\" (UniqueName: \"kubernetes.io/projected/07864a18-dd3e-479f-b255-933dcf240507-kube-api-access-bn948\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756951 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756965 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757043 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.756981 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12a6b3d-1e9d-403f-9108-519c4b4bc898-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757063 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757350 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757368 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757430 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757433 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.757879 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.757733 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.758219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.758066 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:09.759102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759056 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:09.759201 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759148 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:09.759260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759205 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.759384 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759368 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "kube-api-access-nlm8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.759491 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759468 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "kube-api-access-sjx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.759686 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.759666 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy" pod "cf4db37c-1ede-4c62-91cb-befeacd0ff5b" (UID: "cf4db37c-1ede-4c62-91cb-befeacd0ff5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.760657 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.760641 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:09.760744 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.760731 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy" pod "02c8d3bc-75c6-46a0-bfd5-c458128904de" (UID: "02c8d3bc-75c6-46a0-bfd5-c458128904de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858043 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858071 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858081 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858089 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858099 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858108 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858117 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858125 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858134 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c8d3bc-75c6-46a0-bfd5-c458128904de-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858143 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858151 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858159 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858167 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlm8q\" (UniqueName: \"kubernetes.io/projected/02c8d3bc-75c6-46a0-bfd5-c458128904de-kube-api-access-nlm8q\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858177 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858184 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjx2f\" (UniqueName: \"kubernetes.io/projected/cf4db37c-1ede-4c62-91cb-befeacd0ff5b-kube-api-access-sjx2f\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.858501 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:09.858194 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c8d3bc-75c6-46a0-bfd5-c458128904de-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:10.409017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.408986 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" Nov 03 03:57:10.409017 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.409010 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw" event={"ID":"cf4db37c-1ede-4c62-91cb-befeacd0ff5b","Type":"ContainerDied","Data":"8acdca4d653639fe96b51cd48eea1cff596447d6fe47e358026f633655f76086"} Nov 03 03:57:10.409603 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.409081 5275 scope.go:122] "RemoveContainer" containerID="b6a4f09c49c6aa5e4abafc0298ce0840b3a36cee996fb1d47a71448e23d54d38" Nov 03 03:57:10.427295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.426955 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" event={"ID":"a12a6b3d-1e9d-403f-9108-519c4b4bc898","Type":"ContainerDied","Data":"e74f48f322f2de7dc7c9381e8606aa4d108d96c0108e0a1bb9ebf0c467fb8e39"} Nov 03 03:57:10.427295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.427119 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss" Nov 03 03:57:10.430392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.430013 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" Nov 03 03:57:10.430392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.430022 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77" event={"ID":"02c8d3bc-75c6-46a0-bfd5-c458128904de","Type":"ContainerDied","Data":"15350d22e6146ba46ffb56822b5e5b00b6b72567e92fcc379b6a3f9fd1811cde"} Nov 03 03:57:10.432751 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.432723 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" event={"ID":"07864a18-dd3e-479f-b255-933dcf240507","Type":"ContainerDied","Data":"dcde10ddd78281a7e32e2ab0ea34b09d25cceab19ff832a7c5a2907ba933894f"} Nov 03 03:57:10.432915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.432814 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn" Nov 03 03:57:10.440770 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.440344 5275 scope.go:122] "RemoveContainer" containerID="5c4cf5db2051f8886ffdadcf1b31dc45e12a98f810247177a98125a8fce6ad89" Nov 03 03:57:10.455971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.455941 5275 scope.go:122] "RemoveContainer" containerID="e87cf69df2e35202a6adcf7d6b435b33d95c417b15cf7af7cdea6d170e0ba9f9" Nov 03 03:57:10.466153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.466130 5275 scope.go:122] "RemoveContainer" containerID="a9d52b77856023bf2535fe9124cbb1f8bd8df71b7b23ed2a299ee86fb21a1a4f" Nov 03 03:57:10.517862 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.517808 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w"] Nov 03 03:57:10.520585 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.520562 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-158d23-j8s2w"] Nov 03 03:57:10.577818 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.577770 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:57:10.580950 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.580921 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-tvg77"] Nov 03 03:57:10.695812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.695713 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:57:10.700377 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.700344 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rdpnw"] Nov 03 03:57:10.731041 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.730999 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:57:10.731367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.731311 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" containerID="cri-o://9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa" gracePeriod=30 Nov 03 03:57:10.805144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.805113 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:57:10.809103 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.809076 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8nxmn"] Nov 03 03:57:10.920398 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.920362 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:57:10.924757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.924726 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8x9ss"] Nov 03 03:57:10.985737 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.985643 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" path="/var/lib/kubelet/pods/02c8d3bc-75c6-46a0-bfd5-c458128904de/volumes" Nov 03 03:57:10.986270 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.986249 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07864a18-dd3e-479f-b255-933dcf240507" path="/var/lib/kubelet/pods/07864a18-dd3e-479f-b255-933dcf240507/volumes" Nov 03 03:57:10.986709 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.986690 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989a28ca-7911-44f3-a658-35a98513088a" path="/var/lib/kubelet/pods/989a28ca-7911-44f3-a658-35a98513088a/volumes" Nov 03 03:57:10.987482 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.987460 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" path="/var/lib/kubelet/pods/a12a6b3d-1e9d-403f-9108-519c4b4bc898/volumes" Nov 03 03:57:10.987954 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:10.987935 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" path="/var/lib/kubelet/pods/cf4db37c-1ede-4c62-91cb-befeacd0ff5b/volumes" Nov 03 03:57:12.013922 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.013890 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:57:12.014464 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014449 5275 cpu_manager.go:414] "containerMap: removing container" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" Nov 03 03:57:12.014514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014465 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" Nov 03 03:57:12.014514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014476 5275 cpu_manager.go:414] "containerMap: removing container" podUID="989a28ca-7911-44f3-a658-35a98513088a" containerName="revoke-keys" Nov 03 03:57:12.014514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014483 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989a28ca-7911-44f3-a658-35a98513088a" containerName="revoke-keys" Nov 03 03:57:12.014514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014499 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:12.014514 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014503 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014517 5275 cpu_manager.go:414] "containerMap: removing container" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014522 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014531 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014536 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014550 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014554 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014568 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014572 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014580 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014584 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014598 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" Nov 03 03:57:12.014662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014602 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014676 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5104b7b-3513-4c05-9afe-a77545cc7b9f" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014684 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4db37c-1ede-4c62-91cb-befeacd0ff5b" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014689 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014696 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8485418b-d237-4fcd-9515-f5a70391fbed" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014703 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c8d3bc-75c6-46a0-bfd5-c458128904de" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014709 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="989a28ca-7911-44f3-a658-35a98513088a" containerName="revoke-keys" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014717 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="07864a18-dd3e-479f-b255-933dcf240507" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014724 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d5cdee-ba5f-4169-ac86-4a10114549fd" containerName="istio-proxy" Nov 03 03:57:12.015060 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.014731 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c1bf63-8b06-4b16-b844-74d97525a9c9" containerName="istio-proxy" Nov 03 03:57:12.019758 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.019737 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.072801 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.072765 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d7698f-openshift-default-dockercfg-fvk7t\"" Nov 03 03:57:12.090824 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.090784 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:57:12.199720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.199686 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.199975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.199780 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.199975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.199914 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.199975 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.200048 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200093 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.200068 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.200096 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.200173 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.200268 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.200209 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pvx\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301680 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301643 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301687 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301722 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301744 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301778 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301827 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.301945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301894 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pvx\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.301987 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302057 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302185 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302359 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302458 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302514 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.302806 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.302668 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.304547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.304520 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.304663 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.304642 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.365239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.365209 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pvx\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.378743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.378703 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\") pod \"e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:12.633864 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:12.633756 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:13.072129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.072103 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:57:13.073385 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:57:13.073359 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ce106e_1fc3_4181_a2f4_0e5dbb803756.slice/crio-2a6a8071eb46aa3af7b382a23798c7b7d4eca13b0fc09381a4c5b55aba33bf0c WatchSource:0}: Error finding container 2a6a8071eb46aa3af7b382a23798c7b7d4eca13b0fc09381a4c5b55aba33bf0c: Status 404 returned error can't find the container with id 2a6a8071eb46aa3af7b382a23798c7b7d4eca13b0fc09381a4c5b55aba33bf0c Nov 03 03:57:13.076489 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.076456 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:57:13.076556 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.076525 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:57:13.076599 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.076556 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:57:13.458682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.458655 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:57:13.460358 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.460326 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" event={"ID":"12ce106e-1fc3-4181-a2f4-0e5dbb803756","Type":"ContainerStarted","Data":"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75"} Nov 03 03:57:13.460524 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.460387 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" event={"ID":"12ce106e-1fc3-4181-a2f4-0e5dbb803756","Type":"ContainerStarted","Data":"2a6a8071eb46aa3af7b382a23798c7b7d4eca13b0fc09381a4c5b55aba33bf0c"} Nov 03 03:57:13.461986 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.461960 5275 generic.go:356] "Generic (PLEG): container finished" podID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerID="1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2" exitCode=0 Nov 03 03:57:13.462099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.461998 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" event={"ID":"64ab3527-cce4-4a60-952a-550e02ad7d34","Type":"ContainerDied","Data":"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2"} Nov 03 03:57:13.462099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.462009 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" Nov 03 03:57:13.462099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.462041 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp" event={"ID":"64ab3527-cce4-4a60-952a-550e02ad7d34","Type":"ContainerDied","Data":"40c9c162b830adb4c7b6bf659533cec716235d31ba45778c69396b86a6022838"} Nov 03 03:57:13.462099 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.462071 5275 scope.go:122] "RemoveContainer" containerID="1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2" Nov 03 03:57:13.472554 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.472528 5275 scope.go:122] "RemoveContainer" containerID="1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2" Nov 03 03:57:13.477540 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:57:13.473527 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2\": container with ID starting with 1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2 not found: ID does not exist" containerID="1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2" Nov 03 03:57:13.477540 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.473577 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2"} err="failed to get container status \"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2\": rpc error: code = NotFound desc = could not find container \"1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2\": container with ID starting with 1ccfe6249b6cc3ccfd915c1ec3abb32e6f8c88ec90fedce379b1b300cf6ac3a2 not found: ID does not exist" Nov 03 03:57:13.623586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623498 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623552 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623607 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623670 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623700 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623765 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623764 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:13.623822 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623800 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.624187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.623902 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.624187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.624078 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\") pod \"64ab3527-cce4-4a60-952a-550e02ad7d34\" (UID: \"64ab3527-cce4-4a60-952a-550e02ad7d34\") " Nov 03 03:57:13.624187 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.624152 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:13.624356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.624198 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:13.624569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.624533 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:13.625647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.625040 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.625647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.625071 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.625647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.625086 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.625647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.625100 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ab3527-cce4-4a60-952a-550e02ad7d34-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.625647 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.625107 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:13.626202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.626177 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:13.626502 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.626483 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:13.626660 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.626633 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "kube-api-access-xnnmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:13.627058 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.627039 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo" pod "64ab3527-cce4-4a60-952a-550e02ad7d34" (UID: "64ab3527-cce4-4a60-952a-550e02ad7d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:13.633915 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.633894 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:13.703724 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.703659 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" podStartSLOduration=2.703639053 podStartE2EDuration="2.703639053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:57:13.699632996 +0000 UTC m=+3324.852321484" watchObservedRunningTime="2026-11-03 03:57:13.703639053 +0000 UTC m=+3324.856327475" Nov 03 03:57:13.726576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.726533 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.726576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.726570 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.726576 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.726579 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.726832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.726590 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnnmz\" (UniqueName: \"kubernetes.io/projected/64ab3527-cce4-4a60-952a-550e02ad7d34-kube-api-access-xnnmz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.726832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.726600 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ab3527-cce4-4a60-952a-550e02ad7d34-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:13.887637 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.887551 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:57:13.892254 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:13.892219 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-r8xnp"] Nov 03 03:57:14.640558 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:14.640531 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:14.984893 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:14.984781 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" path="/var/lib/kubelet/pods/64ab3527-cce4-4a60-952a-550e02ad7d34/volumes" Nov 03 03:57:15.478129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:15.478098 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:15.479239 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:15.479218 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:57:16.029216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.029191 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:57:16.051434 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051394 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051501 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051571 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051626 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051603 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051637 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051652 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:16.051803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051672 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.051803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051758 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.052042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051817 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.052042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051882 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:16.052042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.051904 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\") pod \"45a8676c-859c-4956-901e-0408eb4e6e9a\" (UID: \"45a8676c-859c-4956-901e-0408eb4e6e9a\") " Nov 03 03:57:16.052206 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052101 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:16.052414 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052389 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.052497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052419 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a8676c-859c-4956-901e-0408eb4e6e9a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.052497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052435 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.052612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052498 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:16.052702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.052677 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:16.054212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.054147 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:16.054628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.054603 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:57:16.054891 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.054838 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:57:16.055292 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.055263 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z" pod "45a8676c-859c-4956-901e-0408eb4e6e9a" (UID: "45a8676c-859c-4956-901e-0408eb4e6e9a"). InnerVolumeSpecName "kube-api-access-r888z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:16.153969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.153922 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r888z\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-kube-api-access-r888z\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.153969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.153955 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.153969 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.153969 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.154295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.153982 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.154295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.153996 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.154295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.154008 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a8676c-859c-4956-901e-0408eb4e6e9a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:16.487424 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.487391 5275 generic.go:356] "Generic (PLEG): container finished" podID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerID="9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa" exitCode=0 Nov 03 03:57:16.487651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.487447 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" Nov 03 03:57:16.487651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.487468 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerDied","Data":"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa"} Nov 03 03:57:16.487651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.487526 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7" event={"ID":"45a8676c-859c-4956-901e-0408eb4e6e9a","Type":"ContainerDied","Data":"89248b699673acd1682ad9fc877dacf76c44eacf67d465535da383bc7eb7b365"} Nov 03 03:57:16.487651 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.487550 5275 scope.go:122] "RemoveContainer" containerID="9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa" Nov 03 03:57:16.498740 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.498720 5275 scope.go:122] "RemoveContainer" containerID="b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522" Nov 03 03:57:16.511065 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.511041 5275 scope.go:122] "RemoveContainer" containerID="9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa" Nov 03 03:57:16.511409 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:57:16.511373 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa\": container with ID starting with 9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa not found: ID does not exist" containerID="9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa" Nov 03 03:57:16.511520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.511421 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa"} err="failed to get container status \"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa\": rpc error: code = NotFound desc = could not find container \"9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa\": container with ID starting with 9856cc7bbb1814d3781b46e740269e6fef05cd66083bbb7b6bff294e030d06fa not found: ID does not exist" Nov 03 03:57:16.511520 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.511446 5275 scope.go:122] "RemoveContainer" containerID="b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522" Nov 03 03:57:16.511734 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:57:16.511716 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522\": container with ID starting with b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522 not found: ID does not exist" containerID="b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522" Nov 03 03:57:16.511814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.511745 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522"} err="failed to get container status \"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522\": rpc error: code = NotFound desc = could not find container \"b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522\": container with ID starting with b9c989565847ded841134634384f48ff1e64d4bca3dbf09d53bd212636251522 not found: ID does not exist" Nov 03 03:57:16.655606 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.655563 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:57:16.702874 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.702827 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-158d23-openshift-default-5ddc8ff469-4x2q7"] Nov 03 03:57:16.996381 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:16.996345 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" path="/var/lib/kubelet/pods/45a8676c-859c-4956-901e-0408eb4e6e9a/volumes" Nov 03 03:57:19.179362 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.179323 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 03:57:19.179944 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.179928 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:19.180026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.179945 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:19.180026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.179955 5275 cpu_manager.go:414] "containerMap: removing container" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:57:19.180026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.179959 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:57:19.180026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180004 5275 cpu_manager.go:414] "containerMap: removing container" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" Nov 03 03:57:19.180026 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180010 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" Nov 03 03:57:19.180182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180093 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12a6b3d-1e9d-403f-9108-519c4b4bc898" containerName="istio-proxy" Nov 03 03:57:19.180182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180103 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ab3527-cce4-4a60-952a-550e02ad7d34" containerName="istio-proxy" Nov 03 03:57:19.180182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180109 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:57:19.180182 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.180118 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:57:19.184635 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.184614 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.239006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.238975 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7ed2d6\"/\"openshift-service-ca.crt\"" Nov 03 03:57:19.239006 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.239004 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7ed2d6\"/\"kube-root-ca.crt\"" Nov 03 03:57:19.288469 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.288424 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7ed2d6\"/\"default-dockercfg-c27mg\"" Nov 03 03:57:19.288645 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.288547 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7ed2d6\"/\"auth-test-model-7ed2d6-kserve-self-signed-certs\"" Nov 03 03:57:19.289107 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289086 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.289216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289117 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.289216 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289154 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5thk\" (UniqueName: \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.289427 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289333 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.289579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289555 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.289579 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.289605 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.292252 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.292227 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 03:57:19.390894 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.390840 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.390911 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.390933 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.390956 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.390993 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5thk\" (UniqueName: \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391067 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.391041 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.391295 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.391386 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.391503 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.391421 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.393393 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.393376 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.393549 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.393533 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.453077 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.453005 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5thk\" (UniqueName: \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\") pod \"auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.498033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.497986 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:19.879567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:19.879540 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 03:57:19.882347 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:57:19.882311 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec54fe2_3251_4ab7_ac22_4021dc21f032.slice/crio-e2ca414c588e1752f1b78e3db4c304458806bf0fa279a455ca31dac758839e02 WatchSource:0}: Error finding container e2ca414c588e1752f1b78e3db4c304458806bf0fa279a455ca31dac758839e02: Status 404 returned error can't find the container with id e2ca414c588e1752f1b78e3db4c304458806bf0fa279a455ca31dac758839e02 Nov 03 03:57:20.521106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:20.521067 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" event={"ID":"aec54fe2-3251-4ab7-ac22-4021dc21f032","Type":"ContainerStarted","Data":"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e"} Nov 03 03:57:20.521106 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:20.521115 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" event={"ID":"aec54fe2-3251-4ab7-ac22-4021dc21f032","Type":"ContainerStarted","Data":"e2ca414c588e1752f1b78e3db4c304458806bf0fa279a455ca31dac758839e02"} Nov 03 03:57:20.641336 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:20.641282 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" podStartSLOduration=1.641267373 podStartE2EDuration="1.641267373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:57:20.639577121 +0000 UTC m=+3331.792265543" watchObservedRunningTime="2026-11-03 03:57:20.641267373 +0000 UTC m=+3331.793955794" Nov 03 03:57:21.499195 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:21.499158 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:29.499325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:29.499231 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:29.512488 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:29.512462 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:29.594244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:29.594213 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 03:57:32.329248 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:57:32.329211 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:57:52.945085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:57:52.945050 5275 scope.go:122] "RemoveContainer" containerID="d19e9bd6cd675f86ca86ff793be7a6725eb9bd422ebd761c5044bb91bfa8b493" Nov 03 03:58:02.332981 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:58:02.332943 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:58:13.651438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.651398 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:58:13.651942 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.651605 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" containerID="cri-o://fb1a4c5bf4d0ca51e7b10b325f8a48c9db9bab5ffa326a78a96c40d7a469bddd" gracePeriod=30 Nov 03 03:58:13.901909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.901806 5275 generic.go:356] "Generic (PLEG): container finished" podID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerID="fb1a4c5bf4d0ca51e7b10b325f8a48c9db9bab5ffa326a78a96c40d7a469bddd" exitCode=0 Nov 03 03:58:13.901909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.901886 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" event={"ID":"d35439b1-b151-439c-ab4e-a6c0f986d059","Type":"ContainerDied","Data":"fb1a4c5bf4d0ca51e7b10b325f8a48c9db9bab5ffa326a78a96c40d7a469bddd"} Nov 03 03:58:13.936098 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.936070 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:58:13.979781 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.979745 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.980020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.979977 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.980420 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980348 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:58:13.981994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980392 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.981994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980804 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.981994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980879 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.981994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980947 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.981994 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.980975 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\") pod \"d35439b1-b151-439c-ab4e-a6c0f986d059\" (UID: \"d35439b1-b151-439c-ab4e-a6c0f986d059\") " Nov 03 03:58:13.983283 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.983251 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:13.983408 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.983377 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-ca-configmap\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:13.984945 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.984909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "kube-api-access-xffc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:13.985794 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.985755 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:58:13.986347 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.986323 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:58:13.986449 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.986359 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:58:13.986991 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:13.986959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs" pod "d35439b1-b151-439c-ab4e-a6c0f986d059" (UID: "d35439b1-b151-439c-ab4e-a6c0f986d059"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:14.084769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084725 5275 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-cacerts\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.084769 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084771 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-csr-dns-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.085088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084788 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.085088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084802 5275 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d35439b1-b151-439c-ab4e-a6c0f986d059-istio-kubeconfig\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.085088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084816 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xffc7\" (UniqueName: \"kubernetes.io/projected/d35439b1-b151-439c-ab4e-a6c0f986d059-kube-api-access-xffc7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.085088 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.084830 5275 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35439b1-b151-439c-ab4e-a6c0f986d059-local-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:14.911628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.911587 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" event={"ID":"d35439b1-b151-439c-ab4e-a6c0f986d059","Type":"ContainerDied","Data":"550e6d149f3fe74f1e2b5dd302d1beb6abd02db7eb65325290e5cc7ac2d8795a"} Nov 03 03:58:14.911628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.911623 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x" Nov 03 03:58:14.912235 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:14.911662 5275 scope.go:122] "RemoveContainer" containerID="fb1a4c5bf4d0ca51e7b10b325f8a48c9db9bab5ffa326a78a96c40d7a469bddd" Nov 03 03:58:15.126690 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:15.126649 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:58:15.162128 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:15.162030 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fdr5x"] Nov 03 03:58:16.985633 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:16.985593 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" path="/var/lib/kubelet/pods/d35439b1-b151-439c-ab4e-a6c0f986d059/volumes" Nov 03 03:58:17.995628 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.995595 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:58:17.996186 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.996171 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" Nov 03 03:58:17.996237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.996187 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" Nov 03 03:58:17.996237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.996206 5275 cpu_manager.go:414] "containerMap: removing container" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:58:17.996237 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.996211 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a8676c-859c-4956-901e-0408eb4e6e9a" containerName="istio-proxy" Nov 03 03:58:17.996334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:17.996300 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35439b1-b151-439c-ab4e-a6c0f986d059" containerName="discovery" Nov 03 03:58:18.001129 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.001112 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.056571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.056538 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:58:18.102669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.102630 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d7698f\"/\"openshift-service-ca.crt\"" Nov 03 03:58:18.102669 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.102685 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d7698f\"/\"kube-root-ca.crt\"" Nov 03 03:58:18.102959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.102687 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d7698f\"/\"default-dockercfg-plq97\"" Nov 03 03:58:18.102959 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.102718 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d7698f\"/\"noten-model-d7698f-kserve-self-signed-certs\"" Nov 03 03:58:18.134291 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134247 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.134519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134319 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.134519 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134445 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.134655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134556 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.134655 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134598 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.134763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.134655 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9nz\" (UniqueName: \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.235745 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.235699 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.235984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.235759 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.235984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.235798 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.235984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.235917 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.235984 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.235951 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.236227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.236000 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9nz\" (UniqueName: \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.236227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.236196 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.236227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.236205 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.236382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.236280 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.238380 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.238357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.238795 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.238773 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.297271 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.297236 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9nz\" (UniqueName: \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\") pod \"noten-model-d7698f-kserve-6b9fbf58f7-rz7mn\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.313547 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.313518 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:18.701569 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.701542 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:58:18.702966 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:58:18.702940 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod681e241d_60c5_44cf_8a6c_e785ca50e9b2.slice/crio-92e9718b7dbd93ef55e2673ff75a383acff05aa889877e03d6c45daf8bee9fd7 WatchSource:0}: Error finding container 92e9718b7dbd93ef55e2673ff75a383acff05aa889877e03d6c45daf8bee9fd7: Status 404 returned error can't find the container with id 92e9718b7dbd93ef55e2673ff75a383acff05aa889877e03d6c45daf8bee9fd7 Nov 03 03:58:18.941188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.941148 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" event={"ID":"681e241d-60c5-44cf-8a6c-e785ca50e9b2","Type":"ContainerStarted","Data":"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437"} Nov 03 03:58:18.941188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:18.941202 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" event={"ID":"681e241d-60c5-44cf-8a6c-e785ca50e9b2","Type":"ContainerStarted","Data":"92e9718b7dbd93ef55e2673ff75a383acff05aa889877e03d6c45daf8bee9fd7"} Nov 03 03:58:19.059662 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:19.059605 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" podStartSLOduration=2.05959106 podStartE2EDuration="2.05959106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:58:19.057517434 +0000 UTC m=+3390.210205856" watchObservedRunningTime="2026-11-03 03:58:19.05959106 +0000 UTC m=+3390.212279481" Nov 03 03:58:19.314433 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:19.314343 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:28.314914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:28.314871 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:28.327940 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:28.327912 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:29.025005 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:29.024974 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:58:32.334232 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:58:32.334198 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:58:33.717802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.717768 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 03:58:33.722096 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.722073 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.776963 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.776619 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9daccc\"/\"openshift-service-ca.crt\"" Nov 03 03:58:33.822274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822236 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.822274 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822275 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7f7\" (UniqueName: \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.822612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822307 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.822612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822439 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.822612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822480 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.822612 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.822557 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.826523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.826487 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9daccc\"/\"auth-test-model-9daccc-kserve-self-signed-certs\"" Nov 03 03:58:33.826523 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.826494 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9daccc\"/\"kube-root-ca.crt\"" Nov 03 03:58:33.826697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.826562 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9daccc\"/\"default-dockercfg-vmlkx\"" Nov 03 03:58:33.829532 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.829511 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 03:58:33.923571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923531 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.923571 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923572 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7f7\" (UniqueName: \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.923872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923602 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.923872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923635 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.923872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923676 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.923872 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.923700 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.924072 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.924004 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.924143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.924077 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.924200 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.924176 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.926057 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.926040 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.926348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.926332 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:33.985257 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:33.985180 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7f7\" (UniqueName: \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\") pod \"auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:34.034802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:34.034770 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:34.417260 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:34.417233 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 03:58:34.419718 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:58:34.419687 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6953518f_c01f_45d1_bb44_b7cc9bfa9ef8.slice/crio-599a8b21a07fd26357977f6942fd22e0b53a41ac7edf44ef2ea7a680c8ee9d45 WatchSource:0}: Error finding container 599a8b21a07fd26357977f6942fd22e0b53a41ac7edf44ef2ea7a680c8ee9d45: Status 404 returned error can't find the container with id 599a8b21a07fd26357977f6942fd22e0b53a41ac7edf44ef2ea7a680c8ee9d45 Nov 03 03:58:35.055588 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:35.055540 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" event={"ID":"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8","Type":"ContainerStarted","Data":"82d2049da4db0c3d533889406299a4fa67ff89a3e543e8fc6664e65967b1e27a"} Nov 03 03:58:35.056042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:35.055604 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" event={"ID":"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8","Type":"ContainerStarted","Data":"599a8b21a07fd26357977f6942fd22e0b53a41ac7edf44ef2ea7a680c8ee9d45"} Nov 03 03:58:35.174521 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:35.174458 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" podStartSLOduration=2.174440896 podStartE2EDuration="2.174440896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:58:35.17389153 +0000 UTC m=+3406.326579951" watchObservedRunningTime="2026-11-03 03:58:35.174440896 +0000 UTC m=+3406.327129317" Nov 03 03:58:36.035935 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:36.035897 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:36.985288 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:36.985253 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 03:58:36.989092 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:36.989075 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.045325 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.045289 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 03:58:37.059682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.059649 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.059682 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.059696 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.059998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.059720 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.059998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.059826 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttspz\" (UniqueName: \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.059998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.059958 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.060178 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.060000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.090975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.090935 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"default-dockercfg-875mt\"" Nov 03 03:58:37.090975 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.090974 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"auth-test-model-3d8dc5-kserve-self-signed-certs\"" Nov 03 03:58:37.093586 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.093560 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"openshift-service-ca.crt\"" Nov 03 03:58:37.093748 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.093632 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"kube-root-ca.crt\"" Nov 03 03:58:37.160763 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.160732 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.160972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.160809 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.160972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.160867 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.160972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.160903 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttspz\" (UniqueName: \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.160972 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.160950 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.161228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.161007 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.161228 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.161219 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.161366 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.161345 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.161419 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.161389 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.163314 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.163292 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.163561 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.163543 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.222537 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.222507 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttspz\" (UniqueName: \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\") pod \"auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.302531 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.302497 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:37.690392 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:37.690368 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 03:58:37.691955 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:58:37.691922 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13b2efa7_9f97_44c3_83d9_888c4828d2d7.slice/crio-b45116c4e7b79073a42a73fc53eef20577e4c7bff6d2adb5149ca87ecdc49292 WatchSource:0}: Error finding container b45116c4e7b79073a42a73fc53eef20577e4c7bff6d2adb5149ca87ecdc49292: Status 404 returned error can't find the container with id b45116c4e7b79073a42a73fc53eef20577e4c7bff6d2adb5149ca87ecdc49292 Nov 03 03:58:38.077305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:38.077262 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" event={"ID":"13b2efa7-9f97-44c3-83d9-888c4828d2d7","Type":"ContainerStarted","Data":"b95bc3b5274046fee7a06eee12e0b8737ccd19849151a976a49dabc440d645be"} Nov 03 03:58:38.077305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:38.077309 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" event={"ID":"13b2efa7-9f97-44c3-83d9-888c4828d2d7","Type":"ContainerStarted","Data":"b45116c4e7b79073a42a73fc53eef20577e4c7bff6d2adb5149ca87ecdc49292"} Nov 03 03:58:38.194564 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:38.194492 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" podStartSLOduration=2.194475701 podStartE2EDuration="2.194475701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:58:38.193968604 +0000 UTC m=+3409.346657025" watchObservedRunningTime="2026-11-03 03:58:38.194475701 +0000 UTC m=+3409.347164122" Nov 03 03:58:38.303396 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:38.303363 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:40.236049 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.236006 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 03:58:40.240109 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.240084 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.295014 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.294733 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-286a8c\"/\"default-dockercfg-nkr4d\"" Nov 03 03:58:40.299580 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.299548 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 03:58:40.306720 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.306677 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.306928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.306731 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.307020 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.306979 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.307110 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.307084 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.307169 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.307125 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqrk\" (UniqueName: \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.307244 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.307225 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.344702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.344663 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-286a8c\"/\"auth-test-model-286a8c-kserve-self-signed-certs\"" Nov 03 03:58:40.344928 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.344802 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-286a8c\"/\"kube-root-ca.crt\"" Nov 03 03:58:40.345016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.344937 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-286a8c\"/\"openshift-service-ca.crt\"" Nov 03 03:58:40.408295 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408258 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408321 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408422 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408487 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408514 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqrk\" (UniqueName: \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408693 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408557 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408693 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408802 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408734 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.408973 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.408903 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.411085 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.411062 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.411428 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.411410 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.470033 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.470000 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqrk\" (UniqueName: \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\") pod \"auth-test-model-286a8c-kserve-59b596645-5xj5f\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.556451 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.556410 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:40.947378 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:40.947349 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 03:58:40.948370 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:58:40.948343 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2faa216_ec0c_4e1b_a201_20f489fe3406.slice/crio-b656ff32a6da6712952a8105caa5520e495202cdde392f43f1ce6eeb9ab1c9b8 WatchSource:0}: Error finding container b656ff32a6da6712952a8105caa5520e495202cdde392f43f1ce6eeb9ab1c9b8: Status 404 returned error can't find the container with id b656ff32a6da6712952a8105caa5520e495202cdde392f43f1ce6eeb9ab1c9b8 Nov 03 03:58:41.098034 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:41.097993 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" event={"ID":"e2faa216-ec0c-4e1b-a201-20f489fe3406","Type":"ContainerStarted","Data":"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36"} Nov 03 03:58:41.098224 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:41.098050 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" event={"ID":"e2faa216-ec0c-4e1b-a201-20f489fe3406","Type":"ContainerStarted","Data":"b656ff32a6da6712952a8105caa5520e495202cdde392f43f1ce6eeb9ab1c9b8"} Nov 03 03:58:41.215114 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:41.215011 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" podStartSLOduration=1.214994347 podStartE2EDuration="1.214994347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:58:41.214131281 +0000 UTC m=+3412.366819704" watchObservedRunningTime="2026-11-03 03:58:41.214994347 +0000 UTC m=+3412.367682772" Nov 03 03:58:41.557405 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:41.557372 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:44.035653 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.035623 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:44.048883 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.048829 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:44.133155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.133120 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 03:58:44.701382 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.701292 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:58:44.885298 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.885264 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:58:44.885536 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:44.885511 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerName="istio-proxy" containerID="cri-o://9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75" gracePeriod=30 Nov 03 03:58:45.480591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:45.480552 5275 patch_prober.go:29] interesting pod/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:58:45.481131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:45.480619 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:58:47.303212 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:47.303173 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:47.316315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:47.316286 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:48.163010 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:48.162978 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 03:58:50.161438 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.161353 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:58:50.168697 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.168669 5275 generic.go:356] "Generic (PLEG): container finished" podID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerID="9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75" exitCode=0 Nov 03 03:58:50.168886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.168721 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" Nov 03 03:58:50.168886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.168752 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" event={"ID":"12ce106e-1fc3-4181-a2f4-0e5dbb803756","Type":"ContainerDied","Data":"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75"} Nov 03 03:58:50.168886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.168839 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2" event={"ID":"12ce106e-1fc3-4181-a2f4-0e5dbb803756","Type":"ContainerDied","Data":"2a6a8071eb46aa3af7b382a23798c7b7d4eca13b0fc09381a4c5b55aba33bf0c"} Nov 03 03:58:50.168886 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.168873 5275 scope.go:122] "RemoveContainer" containerID="9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75" Nov 03 03:58:50.180136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.180117 5275 scope.go:122] "RemoveContainer" containerID="9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75" Nov 03 03:58:50.180432 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:58:50.180415 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75\": container with ID starting with 9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75 not found: ID does not exist" containerID="9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75" Nov 03 03:58:50.180483 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.180444 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75"} err="failed to get container status \"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75\": rpc error: code = NotFound desc = could not find container \"9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75\": container with ID starting with 9c341b1865e72b0f9cd1f36cd6eb2af8e30362e5c025fa2cbe5161fb51d24b75 not found: ID does not exist" Nov 03 03:58:50.354581 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354549 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.354791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354627 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.354791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354651 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.354791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354705 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.354791 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354771 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.355152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354889 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.355152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354906 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:50.355152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.354973 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.355152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.355007 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.355152 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.355043 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\") pod \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\" (UID: \"12ce106e-1fc3-4181-a2f4-0e5dbb803756\") " Nov 03 03:58:50.355567 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.355545 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.356171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.355767 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:50.356171 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.356124 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:58:50.356348 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.356336 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:50.356512 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.356486 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:50.357702 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.357677 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:50.358812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.358712 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:50.358812 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.358777 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "kube-api-access-88pvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:50.359359 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.359335 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo" pod "12ce106e-1fc3-4181-a2f4-0e5dbb803756" (UID: "12ce106e-1fc3-4181-a2f4-0e5dbb803756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456713 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456745 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456756 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456769 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456779 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88pvx\" (UniqueName: \"kubernetes.io/projected/12ce106e-1fc3-4181-a2f4-0e5dbb803756-kube-api-access-88pvx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456789 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ce106e-1fc3-4181-a2f4-0e5dbb803756-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456798 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.456803 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.456806 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ce106e-1fc3-4181-a2f4-0e5dbb803756-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:50.557315 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.557275 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:50.570474 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.570445 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:50.591172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.590779 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:58:50.594784 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.594757 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d7698f-openshift-default-86584d6c69-wtlt2"] Nov 03 03:58:50.986148 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:50.986104 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" path="/var/lib/kubelet/pods/12ce106e-1fc3-4181-a2f4-0e5dbb803756/volumes" Nov 03 03:58:51.187715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:51.187682 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 03:58:59.723367 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:58:59.723325 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" podUID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" containerName="main" containerID="cri-o://220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437" gracePeriod=45 Nov 03 03:59:00.002946 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.002921 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:59:00.061703 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.061671 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.061952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.061782 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.061952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.061814 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.061952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.061882 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.061952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.061923 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.062188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062001 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\" (UniqueName: \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\") pod \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\" (UID: \"681e241d-60c5-44cf-8a6c-e785ca50e9b2\") " Nov 03 03:59:00.062188 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062146 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:59:00.062555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062495 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:59:00.062555 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062506 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:59:00.062747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062630 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.062747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062704 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.062747 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.062723 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.064743 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.064711 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:59:00.065080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.065058 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:59:00.065080 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.065071 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz" pod "681e241d-60c5-44cf-8a6c-e785ca50e9b2" (UID: "681e241d-60c5-44cf-8a6c-e785ca50e9b2"). InnerVolumeSpecName "kube-api-access-6r9nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:59:00.165214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.165180 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9nz\" (UniqueName: \"kubernetes.io/projected/681e241d-60c5-44cf-8a6c-e785ca50e9b2-kube-api-access-6r9nz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.165214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.165208 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/681e241d-60c5-44cf-8a6c-e785ca50e9b2-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.165214 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.165218 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/681e241d-60c5-44cf-8a6c-e785ca50e9b2-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:00.244279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.244194 5275 generic.go:356] "Generic (PLEG): container finished" podID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" containerID="220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437" exitCode=0 Nov 03 03:59:00.244279 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.244249 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" Nov 03 03:59:00.244454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.244277 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" event={"ID":"681e241d-60c5-44cf-8a6c-e785ca50e9b2","Type":"ContainerDied","Data":"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437"} Nov 03 03:59:00.244454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.244323 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn" event={"ID":"681e241d-60c5-44cf-8a6c-e785ca50e9b2","Type":"ContainerDied","Data":"92e9718b7dbd93ef55e2673ff75a383acff05aa889877e03d6c45daf8bee9fd7"} Nov 03 03:59:00.244454 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.244340 5275 scope.go:122] "RemoveContainer" containerID="220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437" Nov 03 03:59:00.257873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.257836 5275 scope.go:122] "RemoveContainer" containerID="220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437" Nov 03 03:59:00.258232 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:00.258212 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437\": container with ID starting with 220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437 not found: ID does not exist" containerID="220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437" Nov 03 03:59:00.258305 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.258242 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437"} err="failed to get container status \"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437\": rpc error: code = NotFound desc = could not find container \"220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437\": container with ID starting with 220c201073c3ec68a37c37534fb9672c6f1de3d6bd57dfac191acc11cf912437 not found: ID does not exist" Nov 03 03:59:00.408639 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.408595 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:59:00.412591 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.412558 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d7698f/noten-model-d7698f-kserve-6b9fbf58f7-rz7mn"] Nov 03 03:59:00.986516 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:00.986483 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" path="/var/lib/kubelet/pods/681e241d-60c5-44cf-8a6c-e785ca50e9b2/volumes" Nov 03 03:59:02.337051 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:02.337015 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:59:28.087539 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.087503 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7122-2edd4c"] Nov 03 03:59:28.088120 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088104 5275 cpu_manager.go:414] "containerMap: removing container" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerName="istio-proxy" Nov 03 03:59:28.088167 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088121 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerName="istio-proxy" Nov 03 03:59:28.088207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088171 5275 cpu_manager.go:414] "containerMap: removing container" podUID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" containerName="main" Nov 03 03:59:28.088207 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" containerName="main" Nov 03 03:59:28.088301 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088288 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e241d-60c5-44cf-8a6c-e785ca50e9b2" containerName="main" Nov 03 03:59:28.088337 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.088311 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ce106e-1fc3-4181-a2f4-0e5dbb803756" containerName="istio-proxy" Nov 03 03:59:28.091839 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.091819 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:28.121102 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.121066 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mx9\" (UniqueName: \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\") pod \"test-curl-7122-2edd4c\" (UID: \"54e7d00c-b82e-4029-a519-47a7c7a6281d\") " pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:28.146258 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.146226 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-2edd4c"] Nov 03 03:59:28.192262 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.192233 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr9t9\"" Nov 03 03:59:28.222205 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.222165 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mx9\" (UniqueName: \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\") pod \"test-curl-7122-2edd4c\" (UID: \"54e7d00c-b82e-4029-a519-47a7c7a6281d\") " pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:28.282832 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.282794 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mx9\" (UniqueName: \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\") pod \"test-curl-7122-2edd4c\" (UID: \"54e7d00c-b82e-4029-a519-47a7c7a6281d\") " pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:28.403990 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.403885 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:28.782678 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.782651 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-2edd4c"] Nov 03 03:59:28.784027 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:59:28.783998 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54e7d00c_b82e_4029_a519_47a7c7a6281d.slice/crio-5f4e8dc205c05bcf3544b9632fe7e2eb3ba7db6f5a843ecaf30e9b503231e8db WatchSource:0}: Error finding container 5f4e8dc205c05bcf3544b9632fe7e2eb3ba7db6f5a843ecaf30e9b503231e8db: Status 404 returned error can't find the container with id 5f4e8dc205c05bcf3544b9632fe7e2eb3ba7db6f5a843ecaf30e9b503231e8db Nov 03 03:59:28.785842 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:28.785823 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:59:29.476278 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:29.476236 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-2edd4c" event={"ID":"54e7d00c-b82e-4029-a519-47a7c7a6281d","Type":"ContainerStarted","Data":"5f4e8dc205c05bcf3544b9632fe7e2eb3ba7db6f5a843ecaf30e9b503231e8db"} Nov 03 03:59:32.331620 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:32.331582 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 03:59:32.503173 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:32.503135 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-2edd4c" event={"ID":"54e7d00c-b82e-4029-a519-47a7c7a6281d","Type":"ContainerStarted","Data":"dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c"} Nov 03 03:59:32.618341 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:32.618220 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7122-2edd4c" podStartSLOduration=1.7588659199999999 podStartE2EDuration="4.618203982s" totalImagesPullingTime="2.859338062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:59:32.615897009 +0000 UTC m=+3463.768585429" watchObservedRunningTime="2026-11-03 03:59:32.618203982 +0000 UTC m=+3463.770892402" Nov 03 03:59:34.102985 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.102946 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7122-2edd4c"] Nov 03 03:59:34.109131 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.109096 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7122-2edd4c"] Nov 03 03:59:34.277383 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.277339 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7122-0239e1"] Nov 03 03:59:34.278218 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.278195 5275 cpu_manager.go:414] "containerMap: removing container" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" containerName="test-curl-7122-2edd4c" Nov 03 03:59:34.278339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.278219 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" containerName="test-curl-7122-2edd4c" Nov 03 03:59:34.278406 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.278378 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" containerName="test-curl-7122-2edd4c" Nov 03 03:59:34.282457 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.282432 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:34.335202 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.335168 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-0239e1"] Nov 03 03:59:34.405394 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.405306 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257kx\" (UniqueName: \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\") pod \"test-curl-7122-0239e1\" (UID: \"424ed680-105a-4452-b202-17f21e8cd086\") " pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:34.506654 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.506623 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257kx\" (UniqueName: \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\") pod \"test-curl-7122-0239e1\" (UID: \"424ed680-105a-4452-b202-17f21e8cd086\") " pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:34.518909 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.518864 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7122-2edd4c" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" containerName="test-curl-7122-2edd4c" containerID="cri-o://dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c" gracePeriod=2 Nov 03 03:59:34.567453 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.567416 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257kx\" (UniqueName: \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\") pod \"test-curl-7122-0239e1\" (UID: \"424ed680-105a-4452-b202-17f21e8cd086\") " pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:34.569811 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:34.569774 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-2edd4c\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:34.596139 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.596100 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:34.979257 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:59:34.979224 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424ed680_105a_4452_b202_17f21e8cd086.slice/crio-0787beff4edfffd9c518a40fdae58dbf7f9f8d30b83b0290dc8c9beef220d405 WatchSource:0}: Error finding container 0787beff4edfffd9c518a40fdae58dbf7f9f8d30b83b0290dc8c9beef220d405: Status 404 returned error can't find the container with id 0787beff4edfffd9c518a40fdae58dbf7f9f8d30b83b0290dc8c9beef220d405 Nov 03 03:59:34.987227 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:34.987201 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7122-0239e1"] Nov 03 03:59:35.526728 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:35.526693 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-0239e1" event={"ID":"424ed680-105a-4452-b202-17f21e8cd086","Type":"ContainerStarted","Data":"48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251"} Nov 03 03:59:35.527385 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:35.526741 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7122-0239e1" event={"ID":"424ed680-105a-4452-b202-17f21e8cd086","Type":"ContainerStarted","Data":"0787beff4edfffd9c518a40fdae58dbf7f9f8d30b83b0290dc8c9beef220d405"} Nov 03 03:59:35.642356 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:35.642298 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7122-0239e1" podStartSLOduration=1.642282917 podStartE2EDuration="1.642282917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:59:35.639432397 +0000 UTC m=+3466.792120819" watchObservedRunningTime="2026-11-03 03:59:35.642282917 +0000 UTC m=+3466.794971338" Nov 03 03:59:36.764155 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:36.764125 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:36.835971 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:36.835871 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\" (UniqueName: \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\") pod \"54e7d00c-b82e-4029-a519-47a7c7a6281d\" (UID: \"54e7d00c-b82e-4029-a519-47a7c7a6281d\") " Nov 03 03:59:36.838091 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:36.838062 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9" pod "54e7d00c-b82e-4029-a519-47a7c7a6281d" (UID: "54e7d00c-b82e-4029-a519-47a7c7a6281d"). InnerVolumeSpecName "kube-api-access-86mx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:59:36.937692 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:36.937659 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mx9\" (UniqueName: \"kubernetes.io/projected/54e7d00c-b82e-4029-a519-47a7c7a6281d-kube-api-access-86mx9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:36.985050 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:36.985015 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e7d00c-b82e-4029-a519-47a7c7a6281d" path="/var/lib/kubelet/pods/54e7d00c-b82e-4029-a519-47a7c7a6281d/volumes" Nov 03 03:59:37.229833 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.229737 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7122-0239e1"] Nov 03 03:59:37.234505 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.234475 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7122-0239e1"] Nov 03 03:59:37.542162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.542117 5275 generic.go:356] "Generic (PLEG): container finished" podID="54e7d00c-b82e-4029-a519-47a7c7a6281d" containerID="dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c" exitCode=137 Nov 03 03:59:37.542379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.542171 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-2edd4c" Nov 03 03:59:37.542379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.542278 5275 scope.go:122] "RemoveContainer" containerID="dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c" Nov 03 03:59:37.542379 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.542327 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7122-0239e1" podUID="424ed680-105a-4452-b202-17f21e8cd086" containerName="test-curl-7122-0239e1" containerID="cri-o://48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251" gracePeriod=2 Nov 03 03:59:37.553042 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.553013 5275 scope.go:122] "RemoveContainer" containerID="dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c" Nov 03 03:59:37.553409 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:37.553384 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c\": container with ID starting with dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c not found: ID does not exist" containerID="dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c" Nov 03 03:59:37.553497 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:37.553417 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c"} err="failed to get container status \"dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c\": rpc error: code = NotFound desc = could not find container \"dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c\": container with ID starting with dd761682e40075cd93667b9f13466e38eb382e6ed8a622734c16eefdc5fbbd1c not found: ID does not exist" Nov 03 03:59:37.593770 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:37.593723 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:37.644842 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:37.644804 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:38.568508 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.568474 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 03:59:38.569687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.569657 5275 cpu_manager.go:414] "containerMap: removing container" podUID="424ed680-105a-4452-b202-17f21e8cd086" containerName="test-curl-7122-0239e1" Nov 03 03:59:38.569687 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.569680 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424ed680-105a-4452-b202-17f21e8cd086" containerName="test-curl-7122-0239e1" Nov 03 03:59:38.569873 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.569793 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="424ed680-105a-4452-b202-17f21e8cd086" containerName="test-curl-7122-0239e1" Nov 03 03:59:38.574470 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.574450 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.633136 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.633089 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5172b1-openshift-default-dockercfg-hjxrw\"" Nov 03 03:59:38.658658 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658623 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b8n\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.658917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658664 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.658917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658690 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.658917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658819 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.659135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658929 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.659135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.658989 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.659135 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.659098 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.659313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.659194 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.659313 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.659236 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.684700 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.684656 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:38.757112 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761665 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761729 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761801 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761841 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761911 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b8n\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761946 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.761974 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.762032 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.762116 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.762651 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.763766 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.764158 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.764552 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.765914 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.764669 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.774144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.768461 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.774144 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.770015 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.823924 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.823807 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b8n\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.824998 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.824938 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\") pod \"e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:38.888133 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:38.888088 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:39.033520 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:39.033474 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:39.328614 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.328583 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 03:59:39.332047 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:59:39.332015 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89da8fb_d241_443a_a119_0be1bfef9c2b.slice/crio-578bcc5c09152a3392a84b35a45ebbd20e8af0912dfdc1892914f4feba8df38d WatchSource:0}: Error finding container 578bcc5c09152a3392a84b35a45ebbd20e8af0912dfdc1892914f4feba8df38d: Status 404 returned error can't find the container with id 578bcc5c09152a3392a84b35a45ebbd20e8af0912dfdc1892914f4feba8df38d Nov 03 03:59:39.335841 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.335800 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:59:39.335996 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.335929 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:59:39.336061 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.335994 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 03:59:39.560698 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.560656 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerStarted","Data":"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c"} Nov 03 03:59:39.560912 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.560716 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerStarted","Data":"578bcc5c09152a3392a84b35a45ebbd20e8af0912dfdc1892914f4feba8df38d"} Nov 03 03:59:39.684322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.684259 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podStartSLOduration=1.6842389789999999 podStartE2EDuration="1.684238979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:59:39.681071233 +0000 UTC m=+3470.833759667" watchObservedRunningTime="2026-11-03 03:59:39.684238979 +0000 UTC m=+3470.836927402" Nov 03 03:59:39.810715 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.810637 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:39.877322 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.877216 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\" (UniqueName: \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\") pod \"424ed680-105a-4452-b202-17f21e8cd086\" (UID: \"424ed680-105a-4452-b202-17f21e8cd086\") " Nov 03 03:59:39.879513 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.879480 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx" pod "424ed680-105a-4452-b202-17f21e8cd086" (UID: "424ed680-105a-4452-b202-17f21e8cd086"). InnerVolumeSpecName "kube-api-access-257kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:59:39.888643 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.888616 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:39.978310 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:39.978280 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257kx\" (UniqueName: \"kubernetes.io/projected/424ed680-105a-4452-b202-17f21e8cd086-kube-api-access-257kx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:59:40.572304 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.572265 5275 generic.go:356] "Generic (PLEG): container finished" podID="424ed680-105a-4452-b202-17f21e8cd086" containerID="48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251" exitCode=137 Nov 03 03:59:40.572486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.572366 5275 scope.go:122] "RemoveContainer" containerID="48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251" Nov 03 03:59:40.572486 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.572397 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:40.584339 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.584312 5275 scope.go:122] "RemoveContainer" containerID="48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251" Nov 03 03:59:40.584667 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:40.584639 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251\": container with ID starting with 48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251 not found: ID does not exist" containerID="48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251" Nov 03 03:59:40.584783 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.584678 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251"} err="failed to get container status \"48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251\": rpc error: code = NotFound desc = could not find container \"48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251\": container with ID starting with 48240606d1a3c8050a6577fffaad918daf69106e68d134ab7111a658204a9251 not found: ID does not exist" Nov 03 03:59:40.632556 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:40.632507 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:40.896917 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.896805 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:40.986323 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:40.986292 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424ed680-105a-4452-b202-17f21e8cd086" path="/var/lib/kubelet/pods/424ed680-105a-4452-b202-17f21e8cd086/volumes" Nov 03 03:59:41.118533 ip-10-0-136-7 kubenswrapper[5275]: E1103 03:59:41.118492 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7122-0239e1\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="424ed680-105a-4452-b202-17f21e8cd086" pod="opendatahub/test-curl-7122-0239e1" Nov 03 03:59:41.581143 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:41.581115 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:41.582081 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:41.582059 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 03:59:46.334263 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.334227 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 03:59:46.338461 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.338443 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.392253 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.392220 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5172b1\"" Nov 03 03:59:46.441814 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.441781 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5172b1-dockercfg-l75q7\"" Nov 03 03:59:46.446664 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.446624 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 03:59:46.463757 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.463714 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9gl\" (UniqueName: \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.464220 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.464196 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.494559 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.494521 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 03:59:46.502804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.502705 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.549584 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.549550 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 03:59:46.565510 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.565469 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9gl\" (UniqueName: \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.565725 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.565552 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.566487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.566465 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.626334 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.626264 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9gl\" (UniqueName: \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\") pod \"payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.654123 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.654083 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:46.666867 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.666817 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.667016 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.666990 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72t6m\" (UniqueName: \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.768952 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.768915 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.769164 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.768983 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72t6m\" (UniqueName: \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.769804 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.769775 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:46.829925 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:46.829891 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72t6m\" (UniqueName: \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\") pod \"payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:47.034162 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.034129 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 03:59:47.035893 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:59:47.035862 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e6eebe_0057_4893_adc6_204f5a9d0143.slice/crio-a7008c1c779aad25ffbd3d0b1d9dce064b6ae31a34133097451fd20e9753b678 WatchSource:0}: Error finding container a7008c1c779aad25ffbd3d0b1d9dce064b6ae31a34133097451fd20e9753b678: Status 404 returned error can't find the container with id a7008c1c779aad25ffbd3d0b1d9dce064b6ae31a34133097451fd20e9753b678 Nov 03 03:59:47.121153 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.121124 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:47.509360 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.509332 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 03:59:47.511317 ip-10-0-136-7 kubenswrapper[5275]: W1103 03:59:47.511293 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b788c3_9f2f_4255_a302_c1e80b37449b.slice/crio-e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3 WatchSource:0}: Error finding container e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3: Status 404 returned error can't find the container with id e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3 Nov 03 03:59:47.625306 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.625263 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" event={"ID":"88e6eebe-0057-4893-adc6-204f5a9d0143","Type":"ContainerStarted","Data":"ff7d35ba4f6cd6240a7b9456ae5361a74458e6fe9fc068c7384e73ce2e3796c8"} Nov 03 03:59:47.625487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.625321 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" event={"ID":"88e6eebe-0057-4893-adc6-204f5a9d0143","Type":"ContainerStarted","Data":"a7008c1c779aad25ffbd3d0b1d9dce064b6ae31a34133097451fd20e9753b678"} Nov 03 03:59:47.625487 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.625361 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:47.627172 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.627141 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" event={"ID":"b5b788c3-9f2f-4255-a302-c1e80b37449b","Type":"ContainerStarted","Data":"dde64ffe23359696ca9aad0ad435497ee774e9ca761d83c41e4ddb976fd81b0a"} Nov 03 03:59:47.627309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.627193 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" event={"ID":"b5b788c3-9f2f-4255-a302-c1e80b37449b","Type":"ContainerStarted","Data":"e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3"} Nov 03 03:59:47.627309 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.627256 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 03:59:47.741977 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.741907 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" podStartSLOduration=1.741887594 podStartE2EDuration="1.741887594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:59:47.740709988 +0000 UTC m=+3478.893398421" watchObservedRunningTime="2026-11-03 03:59:47.741887594 +0000 UTC m=+3478.894576016" Nov 03 03:59:47.860125 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:47.860050 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" podStartSLOduration=1.860030472 podStartE2EDuration="1.860030472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:59:47.855420066 +0000 UTC m=+3479.008108489" watchObservedRunningTime="2026-11-03 03:59:47.860030472 +0000 UTC m=+3479.012718897" Nov 03 03:59:58.639219 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:58.639186 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 03:59:58.639656 ip-10-0-136-7 kubenswrapper[5275]: I1103 03:59:58.639258 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 04:00:00.178564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.178523 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:00.185129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.185092 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.189595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.189561 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:00.200921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.200864 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:00.202089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.202063 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.206918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.206893 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.228011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.227985 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:00.232150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.232125 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:00.232296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.232250 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.236221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.236200 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:00.236370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.236336 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.238895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.238870 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.239012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.238931 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7db\" (UniqueName: \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.239054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.239010 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wdf\" (UniqueName: \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.239197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.239180 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.240166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.240146 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:00.240296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.240277 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.244976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.244954 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:00.245099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.245070 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.248993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.248965 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:00.249106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.249052 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.255061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.255032 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:00.257942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.257918 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.260504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.260480 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.274983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.274899 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:00:00.274983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.274925 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:00:00.275162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.274990 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:00:00.282109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.282064 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:00.286574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.286545 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:00.289788 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.289763 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:00.292752 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.292730 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:00.336449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.336113 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:00.338876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.338810 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340550 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ftmm\" (UniqueName: \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340592 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340628 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b695\" (UniqueName: \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340674 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340702 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btq6z\" (UniqueName: \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.340801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340736 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.341241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.340957 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x568p\" (UniqueName: \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.341241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.341014 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmck\" (UniqueName: \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.341738 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.341715 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.341907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.341887 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.341992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.341938 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb54g\" (UniqueName: \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.342058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.341995 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.342058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342045 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.342179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342102 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.342179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342141 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342177 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342207 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vpx\" (UniqueName: \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342240 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7db\" (UniqueName: \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342273 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wdf\" (UniqueName: \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342383 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.342570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.342429 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vrq\" (UniqueName: \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.343148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.343125 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.387746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.387552 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:00.392547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.392510 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:00.404116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.402666 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:00.407429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.407397 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:00.409753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.409728 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7db\" (UniqueName: \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.410713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.410691 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wdf\" (UniqueName: \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.443448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443358 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.443448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443406 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vrq\" (UniqueName: \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.443448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443439 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ftmm\" (UniqueName: \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443459 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443478 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b695\" (UniqueName: \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443517 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btq6z\" (UniqueName: \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443553 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443591 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x568p\" (UniqueName: \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443615 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmck\" (UniqueName: \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443648 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443672 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb54g\" (UniqueName: \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.443750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443703 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.444288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443757 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.444288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443839 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.444288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.443893 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.444288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444184 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.444546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444510 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.444546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444485 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.444673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444555 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.444673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444595 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.444673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.444601 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vpx\" (UniqueName: \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.445141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.445122 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.445211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.445188 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.445246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.445226 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.506100 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.506063 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:00.506681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.506654 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x568p\" (UniqueName: \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.506801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.506774 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb54g\" (UniqueName: \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.507204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.507176 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ftmm\" (UniqueName: \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.507576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.507471 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vpx\" (UniqueName: \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.507576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.507486 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btq6z\" (UniqueName: \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.507576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.507494 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b695\" (UniqueName: \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.507759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.507744 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vrq\" (UniqueName: \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.508804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.508785 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmck\" (UniqueName: \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.518859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.518828 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:00.526203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.526172 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:00.548350 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.548316 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:00.558617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.558436 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:00.577966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.577126 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:00.590454 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.590258 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:00.594786 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.594600 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:00.606252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.605045 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:00.611261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.611047 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:00.926349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:00.926315 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:00.927285 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:00.927256 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52910ab_2950_47db_89f3_b83034339c65.slice/crio-15c8e3b37262b7a3b806291514acad98ff0226c4cc8ffdd80266b83a6a5a6fa3 WatchSource:0}: Error finding container 15c8e3b37262b7a3b806291514acad98ff0226c4cc8ffdd80266b83a6a5a6fa3: Status 404 returned error can't find the container with id 15c8e3b37262b7a3b806291514acad98ff0226c4cc8ffdd80266b83a6a5a6fa3 Nov 03 04:00:01.125686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.125656 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:01.125817 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:01.125791 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a743730_708a_4b42_8ce8_a7ee65693f1e.slice/crio-9de55a605019a0f41b2928403f66012b659180f37617c0f33281b749fe13f039 WatchSource:0}: Error finding container 9de55a605019a0f41b2928403f66012b659180f37617c0f33281b749fe13f039: Status 404 returned error can't find the container with id 9de55a605019a0f41b2928403f66012b659180f37617c0f33281b749fe13f039 Nov 03 04:00:01.175443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.174377 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:01.180531 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:01.178310 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d784e14_958f_48c8_b4e9_4efeaf79855f.slice/crio-fa820dd7e0b16b5a93497c1c7a8e764c92d1cd8ef6d93c7b721d5eee24feb448 WatchSource:0}: Error finding container fa820dd7e0b16b5a93497c1c7a8e764c92d1cd8ef6d93c7b721d5eee24feb448: Status 404 returned error can't find the container with id fa820dd7e0b16b5a93497c1c7a8e764c92d1cd8ef6d93c7b721d5eee24feb448 Nov 03 04:00:01.180531 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:01.178825 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d380252_e591_4f85_8b8d_5d657fb6b8fb.slice/crio-a77c836b12c294c3532395b70132771d65ff384db9a64baccc8f98971e681696 WatchSource:0}: Error finding container a77c836b12c294c3532395b70132771d65ff384db9a64baccc8f98971e681696: Status 404 returned error can't find the container with id a77c836b12c294c3532395b70132771d65ff384db9a64baccc8f98971e681696 Nov 03 04:00:01.180531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.180226 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:01.195907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.192047 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:01.208422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.208397 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:01.244215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.244183 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:01.244822 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:01.244796 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e26e335_dd3c_420f_be37_90fa55b62d31.slice/crio-58fa7c40e58845725ec95dab63f7003f2a4d8eb4a2a14c6de2bbb9eab2f0f646 WatchSource:0}: Error finding container 58fa7c40e58845725ec95dab63f7003f2a4d8eb4a2a14c6de2bbb9eab2f0f646: Status 404 returned error can't find the container with id 58fa7c40e58845725ec95dab63f7003f2a4d8eb4a2a14c6de2bbb9eab2f0f646 Nov 03 04:00:01.245924 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:00:01.245896 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7363712c_e020_4595_81be_6e008381c837.slice/crio-86e31c64242ed7445c6bb8f82e689dc03a13a147706502637723a3844ee6fce5 WatchSource:0}: Error finding container 86e31c64242ed7445c6bb8f82e689dc03a13a147706502637723a3844ee6fce5: Status 404 returned error can't find the container with id 86e31c64242ed7445c6bb8f82e689dc03a13a147706502637723a3844ee6fce5 Nov 03 04:00:01.248493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.248470 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:01.290475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.290386 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:01.293427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.293402 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:01.745045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.745004 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerStarted","Data":"b226d4ed47abe2b1e6789fe9e4677581d0cb0c33d36a7132d3f60430c2ba628a"} Nov 03 04:00:01.745254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.745065 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerStarted","Data":"a77c836b12c294c3532395b70132771d65ff384db9a64baccc8f98971e681696"} Nov 03 04:00:01.746369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.746345 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerStarted","Data":"86e31c64242ed7445c6bb8f82e689dc03a13a147706502637723a3844ee6fce5"} Nov 03 04:00:01.747494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.747470 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerStarted","Data":"58fa7c40e58845725ec95dab63f7003f2a4d8eb4a2a14c6de2bbb9eab2f0f646"} Nov 03 04:00:01.752158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.748863 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerStarted","Data":"85b7d53b645fbd4f3729f4638363f3e944b63ba630d0b908b15929eb6fd76bfe"} Nov 03 04:00:01.752158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.748912 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerStarted","Data":"15c8e3b37262b7a3b806291514acad98ff0226c4cc8ffdd80266b83a6a5a6fa3"} Nov 03 04:00:01.752158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.750540 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerStarted","Data":"9de55a605019a0f41b2928403f66012b659180f37617c0f33281b749fe13f039"} Nov 03 04:00:01.752158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.751838 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerStarted","Data":"fe33060cc5bace90c31d22c173ea48a364bedefc67c88bd4330809af8cdefbd0"} Nov 03 04:00:01.753364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.753341 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerStarted","Data":"fa820dd7e0b16b5a93497c1c7a8e764c92d1cd8ef6d93c7b721d5eee24feb448"} Nov 03 04:00:01.755581 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.755524 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerStarted","Data":"5ebac69e846f1337f2841c74eb5f77ce1085d36a279b03328029561bfbda4ce3"} Nov 03 04:00:01.755685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.755592 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerStarted","Data":"95f107b5e30a92f2eeaedc3f52fb6e45afd8ca37a3abf377c0134c5b98dd8331"} Nov 03 04:00:01.759153 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.758329 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerStarted","Data":"14ad6b4a3df918b6a8a66fd44054c5d63417f8738f355e2bc9c6cf6efbc3d3a2"} Nov 03 04:00:01.759153 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.758373 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerStarted","Data":"e5d9d519387136e50cf5ac8f915356a16282270d9ca13c3274a0850f333e53cd"} Nov 03 04:00:01.760548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.760520 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerStarted","Data":"52d186e7ee9db096fba3b0abc94a8b7c0ac1cdf97b9335807a47f72ee3317006"} Nov 03 04:00:01.866402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.866344 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" podStartSLOduration=1.866328256 podStartE2EDuration="1.866328256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:01.859986156 +0000 UTC m=+3493.012674576" watchObservedRunningTime="2026-11-03 04:00:01.866328256 +0000 UTC m=+3493.019016678" Nov 03 04:00:01.979991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:01.979929 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" podStartSLOduration=1.979910996 podStartE2EDuration="1.979910996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:01.975409812 +0000 UTC m=+3493.128098233" watchObservedRunningTime="2026-11-03 04:00:01.979910996 +0000 UTC m=+3493.132599419" Nov 03 04:00:02.095489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.095437 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" podStartSLOduration=2.095423205 podStartE2EDuration="2.095423205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:02.093607238 +0000 UTC m=+3493.246295662" watchObservedRunningTime="2026-11-03 04:00:02.095423205 +0000 UTC m=+3493.248111654" Nov 03 04:00:02.210251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.210197 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" podStartSLOduration=2.210183225 podStartE2EDuration="2.210183225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:02.208914057 +0000 UTC m=+3493.361602492" watchObservedRunningTime="2026-11-03 04:00:02.210183225 +0000 UTC m=+3493.362871701" Nov 03 04:00:02.331397 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:02.331358 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:00:02.769305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.769273 5275 generic.go:356] "Generic (PLEG): container finished" podID="faf74253-7445-427d-a573-3fd58a0690d3" containerID="5ebac69e846f1337f2841c74eb5f77ce1085d36a279b03328029561bfbda4ce3" exitCode=60 Nov 03 04:00:02.769501 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.769315 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerDied","Data":"5ebac69e846f1337f2841c74eb5f77ce1085d36a279b03328029561bfbda4ce3"} Nov 03 04:00:02.769667 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.769646 5275 scope.go:122] "RemoveContainer" containerID="5ebac69e846f1337f2841c74eb5f77ce1085d36a279b03328029561bfbda4ce3" Nov 03 04:00:02.771082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.771060 5275 generic.go:356] "Generic (PLEG): container finished" podID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerID="14ad6b4a3df918b6a8a66fd44054c5d63417f8738f355e2bc9c6cf6efbc3d3a2" exitCode=60 Nov 03 04:00:02.771182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.771124 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerDied","Data":"14ad6b4a3df918b6a8a66fd44054c5d63417f8738f355e2bc9c6cf6efbc3d3a2"} Nov 03 04:00:02.771469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.771450 5275 scope.go:122] "RemoveContainer" containerID="14ad6b4a3df918b6a8a66fd44054c5d63417f8738f355e2bc9c6cf6efbc3d3a2" Nov 03 04:00:02.772980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.772961 5275 generic.go:356] "Generic (PLEG): container finished" podID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerID="019af1164ac8825ba415ac51a4f7a2027484edd3f685a3a3c6db194841a667c6" exitCode=60 Nov 03 04:00:02.773098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.773032 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerDied","Data":"019af1164ac8825ba415ac51a4f7a2027484edd3f685a3a3c6db194841a667c6"} Nov 03 04:00:02.773321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.773248 5275 scope.go:122] "RemoveContainer" containerID="019af1164ac8825ba415ac51a4f7a2027484edd3f685a3a3c6db194841a667c6" Nov 03 04:00:02.774650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.774630 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerID="b226d4ed47abe2b1e6789fe9e4677581d0cb0c33d36a7132d3f60430c2ba628a" exitCode=60 Nov 03 04:00:02.774750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.774655 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerDied","Data":"b226d4ed47abe2b1e6789fe9e4677581d0cb0c33d36a7132d3f60430c2ba628a"} Nov 03 04:00:02.774932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.774916 5275 scope.go:122] "RemoveContainer" containerID="b226d4ed47abe2b1e6789fe9e4677581d0cb0c33d36a7132d3f60430c2ba628a" Nov 03 04:00:02.776294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.776275 5275 generic.go:356] "Generic (PLEG): container finished" podID="7363712c-e020-4595-81be-6e008381c837" containerID="4e149a20ab1d55e8c0a199fd5df6156a6ee8d4900bf1bc8f2e9aa92a10b42298" exitCode=60 Nov 03 04:00:02.776372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.776354 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerDied","Data":"4e149a20ab1d55e8c0a199fd5df6156a6ee8d4900bf1bc8f2e9aa92a10b42298"} Nov 03 04:00:02.776520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.776506 5275 scope.go:122] "RemoveContainer" containerID="4e149a20ab1d55e8c0a199fd5df6156a6ee8d4900bf1bc8f2e9aa92a10b42298" Nov 03 04:00:02.778025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.778007 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerID="b33551d986a6e7c629e668196b9025c51b2a44eb72def61cde29461a599bfeb6" exitCode=60 Nov 03 04:00:02.778127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.778059 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerDied","Data":"b33551d986a6e7c629e668196b9025c51b2a44eb72def61cde29461a599bfeb6"} Nov 03 04:00:02.778288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.778269 5275 scope.go:122] "RemoveContainer" containerID="b33551d986a6e7c629e668196b9025c51b2a44eb72def61cde29461a599bfeb6" Nov 03 04:00:02.779810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.779793 5275 generic.go:356] "Generic (PLEG): container finished" podID="d52910ab-2950-47db-89f3-b83034339c65" containerID="85b7d53b645fbd4f3729f4638363f3e944b63ba630d0b908b15929eb6fd76bfe" exitCode=60 Nov 03 04:00:02.779919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.779891 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerDied","Data":"85b7d53b645fbd4f3729f4638363f3e944b63ba630d0b908b15929eb6fd76bfe"} Nov 03 04:00:02.780075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.780060 5275 scope.go:122] "RemoveContainer" containerID="85b7d53b645fbd4f3729f4638363f3e944b63ba630d0b908b15929eb6fd76bfe" Nov 03 04:00:02.785083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.785061 5275 generic.go:356] "Generic (PLEG): container finished" podID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerID="70e1a0288c53f6c4d1ea02d4c7e42f4bc2e353563bf89077c1581b3c67647ab0" exitCode=60 Nov 03 04:00:02.785247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.785180 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerDied","Data":"70e1a0288c53f6c4d1ea02d4c7e42f4bc2e353563bf89077c1581b3c67647ab0"} Nov 03 04:00:02.785352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.785330 5275 scope.go:122] "RemoveContainer" containerID="70e1a0288c53f6c4d1ea02d4c7e42f4bc2e353563bf89077c1581b3c67647ab0" Nov 03 04:00:02.787121 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.787101 5275 generic.go:356] "Generic (PLEG): container finished" podID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerID="da3e780ef8b4970e2ae11f960863274b484c8adc0ce083a7dda468d3839ee5f1" exitCode=60 Nov 03 04:00:02.787199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.787183 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerDied","Data":"da3e780ef8b4970e2ae11f960863274b484c8adc0ce083a7dda468d3839ee5f1"} Nov 03 04:00:02.787384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.787367 5275 scope.go:122] "RemoveContainer" containerID="da3e780ef8b4970e2ae11f960863274b484c8adc0ce083a7dda468d3839ee5f1" Nov 03 04:00:02.788927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.788905 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerID="ee36120e16f7f09944bed413b8e8bd6b809dcfb934368ac7a8858a291728b00e" exitCode=60 Nov 03 04:00:02.789025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.788938 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerDied","Data":"ee36120e16f7f09944bed413b8e8bd6b809dcfb934368ac7a8858a291728b00e"} Nov 03 04:00:02.789156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:02.789143 5275 scope.go:122] "RemoveContainer" containerID="ee36120e16f7f09944bed413b8e8bd6b809dcfb934368ac7a8858a291728b00e" Nov 03 04:00:03.798164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.798117 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerStarted","Data":"ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142"} Nov 03 04:00:03.800155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.800131 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerStarted","Data":"fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63"} Nov 03 04:00:03.802105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.802080 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerStarted","Data":"5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7"} Nov 03 04:00:03.803896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.803873 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerStarted","Data":"80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307"} Nov 03 04:00:03.805685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.805662 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerStarted","Data":"7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38"} Nov 03 04:00:03.807631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.807604 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerStarted","Data":"d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf"} Nov 03 04:00:03.809400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.809380 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerStarted","Data":"dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73"} Nov 03 04:00:03.811045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.811024 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerStarted","Data":"0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7"} Nov 03 04:00:03.812702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.812683 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerStarted","Data":"6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3"} Nov 03 04:00:03.814553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:03.814532 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerStarted","Data":"cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427"} Nov 03 04:00:04.169196 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.169145 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" podStartSLOduration=4.169128952 podStartE2EDuration="4.169128952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.168263479 +0000 UTC m=+3495.320951901" watchObservedRunningTime="2026-11-03 04:00:04.169128952 +0000 UTC m=+3495.321817377" Nov 03 04:00:04.284965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.284908 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" podStartSLOduration=4.284893674 podStartE2EDuration="4.284893674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.28253045 +0000 UTC m=+3495.435218872" watchObservedRunningTime="2026-11-03 04:00:04.284893674 +0000 UTC m=+3495.437582094" Nov 03 04:00:04.401705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.401655 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" podStartSLOduration=4.401640868 podStartE2EDuration="4.401640868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.39990333 +0000 UTC m=+3495.552591751" watchObservedRunningTime="2026-11-03 04:00:04.401640868 +0000 UTC m=+3495.554329288" Nov 03 04:00:04.632438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.632328 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" podStartSLOduration=4.632314619 podStartE2EDuration="4.632314619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.631086322 +0000 UTC m=+3495.783774745" watchObservedRunningTime="2026-11-03 04:00:04.632314619 +0000 UTC m=+3495.785003039" Nov 03 04:00:04.746706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.746622 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" podStartSLOduration=4.746602918 podStartE2EDuration="4.746602918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.745660091 +0000 UTC m=+3495.898348509" watchObservedRunningTime="2026-11-03 04:00:04.746602918 +0000 UTC m=+3495.899291338" Nov 03 04:00:04.823660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.823626 5275 generic.go:356] "Generic (PLEG): container finished" podID="faf74253-7445-427d-a573-3fd58a0690d3" containerID="80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307" exitCode=60 Nov 03 04:00:04.824264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.823716 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerDied","Data":"80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307"} Nov 03 04:00:04.824264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.823791 5275 scope.go:122] "RemoveContainer" containerID="5ebac69e846f1337f2841c74eb5f77ce1085d36a279b03328029561bfbda4ce3" Nov 03 04:00:04.824264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.824014 5275 scope.go:122] "RemoveContainer" containerID="80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307" Nov 03 04:00:04.824468 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.824350 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw_odh-ai-gateway-infra(faf74253-7445-427d-a573-3fd58a0690d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" podUID="faf74253-7445-427d-a573-3fd58a0690d3" Nov 03 04:00:04.826102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.826069 5275 generic.go:356] "Generic (PLEG): container finished" podID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerID="7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38" exitCode=60 Nov 03 04:00:04.826223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.826156 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerDied","Data":"7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38"} Nov 03 04:00:04.826491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.826475 5275 scope.go:122] "RemoveContainer" containerID="7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38" Nov 03 04:00:04.826782 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.826759 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk_odh-ai-gateway-infra(094e43d5-02a6-4df5-ba0c-c6db69a02418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" Nov 03 04:00:04.828394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.828372 5275 generic.go:356] "Generic (PLEG): container finished" podID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerID="d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf" exitCode=60 Nov 03 04:00:04.828500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.828429 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerDied","Data":"d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf"} Nov 03 04:00:04.828729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.828713 5275 scope.go:122] "RemoveContainer" containerID="d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf" Nov 03 04:00:04.829158 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.829132 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj_odh-ai-gateway-infra(657ed413-18db-474d-8695-79fe8ea9bbf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" Nov 03 04:00:04.830585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.830562 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerID="dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73" exitCode=60 Nov 03 04:00:04.830688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.830641 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerDied","Data":"dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73"} Nov 03 04:00:04.830901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.830887 5275 scope.go:122] "RemoveContainer" containerID="dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73" Nov 03 04:00:04.831150 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.831128 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x_odh-ai-gateway-infra(8d380252-e591-4f85-8b8d-5d657fb6b8fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" Nov 03 04:00:04.832660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.832637 5275 generic.go:356] "Generic (PLEG): container finished" podID="7363712c-e020-4595-81be-6e008381c837" containerID="0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7" exitCode=60 Nov 03 04:00:04.832768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.832713 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerDied","Data":"0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7"} Nov 03 04:00:04.832987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.832964 5275 scope.go:122] "RemoveContainer" containerID="0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7" Nov 03 04:00:04.833248 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.833230 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm_odh-ai-gateway-infra(7363712c-e020-4595-81be-6e008381c837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" podUID="7363712c-e020-4595-81be-6e008381c837" Nov 03 04:00:04.834877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.834834 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerID="6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3" exitCode=60 Nov 03 04:00:04.834972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.834889 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerDied","Data":"6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3"} Nov 03 04:00:04.835237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.835223 5275 scope.go:122] "RemoveContainer" containerID="6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3" Nov 03 04:00:04.835464 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.835445 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz_odh-ai-gateway-infra(8e26e335-dd3c-420f-be37-90fa55b62d31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" Nov 03 04:00:04.836761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.836741 5275 generic.go:356] "Generic (PLEG): container finished" podID="d52910ab-2950-47db-89f3-b83034339c65" containerID="cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427" exitCode=60 Nov 03 04:00:04.836892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.836873 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerDied","Data":"cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427"} Nov 03 04:00:04.837145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.837122 5275 scope.go:122] "RemoveContainer" containerID="cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427" Nov 03 04:00:04.837342 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.837324 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm_odh-ai-gateway-infra(d52910ab-2950-47db-89f3-b83034339c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" podUID="d52910ab-2950-47db-89f3-b83034339c65" Nov 03 04:00:04.837966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.837949 5275 scope.go:122] "RemoveContainer" containerID="14ad6b4a3df918b6a8a66fd44054c5d63417f8738f355e2bc9c6cf6efbc3d3a2" Nov 03 04:00:04.838882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.838781 5275 generic.go:356] "Generic (PLEG): container finished" podID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerID="ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142" exitCode=60 Nov 03 04:00:04.838962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.838874 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerDied","Data":"ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142"} Nov 03 04:00:04.839100 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.839087 5275 scope.go:122] "RemoveContainer" containerID="ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142" Nov 03 04:00:04.839299 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.839282 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz_odh-ai-gateway-infra(1a743730-708a-4b42-8ce8-a7ee65693f1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" Nov 03 04:00:04.840954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.840934 5275 generic.go:356] "Generic (PLEG): container finished" podID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerID="fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63" exitCode=60 Nov 03 04:00:04.841043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.841000 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerDied","Data":"fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63"} Nov 03 04:00:04.841170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.841155 5275 scope.go:122] "RemoveContainer" containerID="fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63" Nov 03 04:00:04.841385 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.841361 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw_odh-ai-gateway-infra(3a0d19a9-8199-4def-b079-9f66b2c82c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" Nov 03 04:00:04.842820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.842804 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerID="5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7" exitCode=60 Nov 03 04:00:04.843001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.842837 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerDied","Data":"5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7"} Nov 03 04:00:04.843125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.843108 5275 scope.go:122] "RemoveContainer" containerID="5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7" Nov 03 04:00:04.843331 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:04.843312 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw_odh-ai-gateway-infra(8d784e14-958f-48c8-b4e9-4efeaf79855f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" Nov 03 04:00:04.850355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.850333 5275 scope.go:122] "RemoveContainer" containerID="019af1164ac8825ba415ac51a4f7a2027484edd3f685a3a3c6db194841a667c6" Nov 03 04:00:04.862681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.862656 5275 scope.go:122] "RemoveContainer" containerID="b226d4ed47abe2b1e6789fe9e4677581d0cb0c33d36a7132d3f60430c2ba628a" Nov 03 04:00:04.875133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.875111 5275 scope.go:122] "RemoveContainer" containerID="4e149a20ab1d55e8c0a199fd5df6156a6ee8d4900bf1bc8f2e9aa92a10b42298" Nov 03 04:00:04.887801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.887774 5275 scope.go:122] "RemoveContainer" containerID="b33551d986a6e7c629e668196b9025c51b2a44eb72def61cde29461a599bfeb6" Nov 03 04:00:04.899969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.899949 5275 scope.go:122] "RemoveContainer" containerID="85b7d53b645fbd4f3729f4638363f3e944b63ba630d0b908b15929eb6fd76bfe" Nov 03 04:00:04.913661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.913642 5275 scope.go:122] "RemoveContainer" containerID="70e1a0288c53f6c4d1ea02d4c7e42f4bc2e353563bf89077c1581b3c67647ab0" Nov 03 04:00:04.929416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.929394 5275 scope.go:122] "RemoveContainer" containerID="da3e780ef8b4970e2ae11f960863274b484c8adc0ce083a7dda468d3839ee5f1" Nov 03 04:00:04.942618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.942589 5275 scope.go:122] "RemoveContainer" containerID="ee36120e16f7f09944bed413b8e8bd6b809dcfb934368ac7a8858a291728b00e" Nov 03 04:00:04.976871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:04.976797 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" podStartSLOduration=4.976780778 podStartE2EDuration="4.976780778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:04.975712132 +0000 UTC m=+3496.128400554" watchObservedRunningTime="2026-11-03 04:00:04.976780778 +0000 UTC m=+3496.129469199" Nov 03 04:00:05.851575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.851544 5275 scope.go:122] "RemoveContainer" containerID="dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73" Nov 03 04:00:05.852078 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.851912 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x_odh-ai-gateway-infra(8d380252-e591-4f85-8b8d-5d657fb6b8fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" Nov 03 04:00:05.853441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.853415 5275 scope.go:122] "RemoveContainer" containerID="0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7" Nov 03 04:00:05.853756 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.853732 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm_odh-ai-gateway-infra(7363712c-e020-4595-81be-6e008381c837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" podUID="7363712c-e020-4595-81be-6e008381c837" Nov 03 04:00:05.855115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.855084 5275 scope.go:122] "RemoveContainer" containerID="6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3" Nov 03 04:00:05.855381 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.855359 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz_odh-ai-gateway-infra(8e26e335-dd3c-420f-be37-90fa55b62d31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" Nov 03 04:00:05.856748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.856728 5275 scope.go:122] "RemoveContainer" containerID="cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427" Nov 03 04:00:05.857012 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.856988 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm_odh-ai-gateway-infra(d52910ab-2950-47db-89f3-b83034339c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" podUID="d52910ab-2950-47db-89f3-b83034339c65" Nov 03 04:00:05.858466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.858446 5275 scope.go:122] "RemoveContainer" containerID="ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142" Nov 03 04:00:05.858708 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.858685 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz_odh-ai-gateway-infra(1a743730-708a-4b42-8ce8-a7ee65693f1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" Nov 03 04:00:05.860067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.860048 5275 scope.go:122] "RemoveContainer" containerID="fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63" Nov 03 04:00:05.860294 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.860272 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw_odh-ai-gateway-infra(3a0d19a9-8199-4def-b079-9f66b2c82c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" Nov 03 04:00:05.861640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.861616 5275 scope.go:122] "RemoveContainer" containerID="5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7" Nov 03 04:00:05.861856 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.861823 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw_odh-ai-gateway-infra(8d784e14-958f-48c8-b4e9-4efeaf79855f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" Nov 03 04:00:05.863370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.863352 5275 scope.go:122] "RemoveContainer" containerID="80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307" Nov 03 04:00:05.863589 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.863566 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw_odh-ai-gateway-infra(faf74253-7445-427d-a573-3fd58a0690d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" podUID="faf74253-7445-427d-a573-3fd58a0690d3" Nov 03 04:00:05.864913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.864896 5275 scope.go:122] "RemoveContainer" containerID="7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38" Nov 03 04:00:05.865088 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.865069 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk_odh-ai-gateway-infra(094e43d5-02a6-4df5-ba0c-c6db69a02418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" Nov 03 04:00:05.866431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:05.866415 5275 scope.go:122] "RemoveContainer" containerID="d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf" Nov 03 04:00:05.866595 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:05.866579 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj_odh-ai-gateway-infra(657ed413-18db-474d-8695-79fe8ea9bbf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" Nov 03 04:00:14.978174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:14.978135 5275 scope.go:122] "RemoveContainer" containerID="0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7" Nov 03 04:00:14.980690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:14.978227 5275 scope.go:122] "RemoveContainer" containerID="dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73" Nov 03 04:00:14.980690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:14.978341 5275 scope.go:122] "RemoveContainer" containerID="6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3" Nov 03 04:00:14.980690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:14.978416 5275 scope.go:122] "RemoveContainer" containerID="7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38" Nov 03 04:00:15.938395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:15.938305 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerStarted","Data":"566501e206603dd53f976a35c4d67c67e2d4f7703ed20e5cdcaef7eea1c1415a"} Nov 03 04:00:15.940287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:15.940262 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerStarted","Data":"2317f2d897f425421d80a3cb3df29bb63a17b1d76325c29fca6fb3aac23e9cd7"} Nov 03 04:00:15.942069 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:15.942044 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerStarted","Data":"800c35fda0aabfedd76ba3cfe7eb73397e68391af47cf2d3667b99b177e07e51"} Nov 03 04:00:15.944031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:15.944012 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerStarted","Data":"26741d01db225951c9a359df738dbcfd04295c4c15042d78b3aa85234aec9cfd"} Nov 03 04:00:15.977233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:15.977202 5275 scope.go:122] "RemoveContainer" containerID="ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142" Nov 03 04:00:16.953144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.953109 5275 generic.go:356] "Generic (PLEG): container finished" podID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerID="566501e206603dd53f976a35c4d67c67e2d4f7703ed20e5cdcaef7eea1c1415a" exitCode=60 Nov 03 04:00:16.953609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.953180 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerDied","Data":"566501e206603dd53f976a35c4d67c67e2d4f7703ed20e5cdcaef7eea1c1415a"} Nov 03 04:00:16.953609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.953274 5275 scope.go:122] "RemoveContainer" containerID="7417353a37cb5958ac70e7533e71475fefbfe4861805811f3d87236927729a38" Nov 03 04:00:16.953717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.953677 5275 scope.go:122] "RemoveContainer" containerID="566501e206603dd53f976a35c4d67c67e2d4f7703ed20e5cdcaef7eea1c1415a" Nov 03 04:00:16.954095 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:16.954069 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk_odh-ai-gateway-infra(094e43d5-02a6-4df5-ba0c-c6db69a02418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" Nov 03 04:00:16.956567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.956541 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerID="2317f2d897f425421d80a3cb3df29bb63a17b1d76325c29fca6fb3aac23e9cd7" exitCode=60 Nov 03 04:00:16.956688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.956640 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerDied","Data":"2317f2d897f425421d80a3cb3df29bb63a17b1d76325c29fca6fb3aac23e9cd7"} Nov 03 04:00:16.957264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.957012 5275 scope.go:122] "RemoveContainer" containerID="2317f2d897f425421d80a3cb3df29bb63a17b1d76325c29fca6fb3aac23e9cd7" Nov 03 04:00:16.957475 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:16.957452 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x_odh-ai-gateway-infra(8d380252-e591-4f85-8b8d-5d657fb6b8fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" Nov 03 04:00:16.962989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.962967 5275 generic.go:356] "Generic (PLEG): container finished" podID="7363712c-e020-4595-81be-6e008381c837" containerID="800c35fda0aabfedd76ba3cfe7eb73397e68391af47cf2d3667b99b177e07e51" exitCode=60 Nov 03 04:00:16.963115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.963038 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerDied","Data":"800c35fda0aabfedd76ba3cfe7eb73397e68391af47cf2d3667b99b177e07e51"} Nov 03 04:00:16.963468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.963451 5275 scope.go:122] "RemoveContainer" containerID="800c35fda0aabfedd76ba3cfe7eb73397e68391af47cf2d3667b99b177e07e51" Nov 03 04:00:16.963806 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:16.963766 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm_odh-ai-gateway-infra(7363712c-e020-4595-81be-6e008381c837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" podUID="7363712c-e020-4595-81be-6e008381c837" Nov 03 04:00:16.965311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.965290 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerID="26741d01db225951c9a359df738dbcfd04295c4c15042d78b3aa85234aec9cfd" exitCode=60 Nov 03 04:00:16.965424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.965368 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerDied","Data":"26741d01db225951c9a359df738dbcfd04295c4c15042d78b3aa85234aec9cfd"} Nov 03 04:00:16.965758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.965741 5275 scope.go:122] "RemoveContainer" containerID="26741d01db225951c9a359df738dbcfd04295c4c15042d78b3aa85234aec9cfd" Nov 03 04:00:16.966156 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:16.966087 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz_odh-ai-gateway-infra(8e26e335-dd3c-420f-be37-90fa55b62d31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" Nov 03 04:00:16.968228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.968204 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerStarted","Data":"1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257"} Nov 03 04:00:16.968600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.968584 5275 scope.go:122] "RemoveContainer" containerID="1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257" Nov 03 04:00:16.969040 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:16.969015 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz_odh-ai-gateway-infra(1a743730-708a-4b42-8ce8-a7ee65693f1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" Nov 03 04:00:16.977276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:16.977254 5275 scope.go:122] "RemoveContainer" containerID="cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427" Nov 03 04:00:17.019893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.019871 5275 scope.go:122] "RemoveContainer" containerID="dfc537d36172a2c806a1e6fabb4a4aff072ead453ffb58834d32973a1467ea73" Nov 03 04:00:17.040951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.040930 5275 scope.go:122] "RemoveContainer" containerID="0fe92aae9a9b648564a5c6df97fda4e5dc6bb2b1dfd20e30fa44b49fd745cfc7" Nov 03 04:00:17.055098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.055073 5275 scope.go:122] "RemoveContainer" containerID="6b1c7f273c488f7870540cb0ae767447e15fafe92adf0d146375f2a1630fa2e3" Nov 03 04:00:17.071383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.071352 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:17.185111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.185075 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:17.302856 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.302809 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:17.416826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.416787 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:17.978204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.978177 5275 scope.go:122] "RemoveContainer" containerID="80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307" Nov 03 04:00:17.978656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.978263 5275 scope.go:122] "RemoveContainer" containerID="d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf" Nov 03 04:00:17.978656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.978450 5275 scope.go:122] "RemoveContainer" containerID="5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7" Nov 03 04:00:17.978656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.978591 5275 scope.go:122] "RemoveContainer" containerID="fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63" Nov 03 04:00:17.987265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.987238 5275 generic.go:356] "Generic (PLEG): container finished" podID="d52910ab-2950-47db-89f3-b83034339c65" containerID="a9aaa7f49ae7485a672e5834a34e890676c870b87e2aa73aec51243ae5bd963a" exitCode=60 Nov 03 04:00:17.987382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.987297 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerDied","Data":"a9aaa7f49ae7485a672e5834a34e890676c870b87e2aa73aec51243ae5bd963a"} Nov 03 04:00:17.987382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.987358 5275 scope.go:122] "RemoveContainer" containerID="cad0c43e3f0f64ebce50db12c9bf89104cf755cc2cc2ac103cb26193ca1d7427" Nov 03 04:00:17.987721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.987706 5275 scope.go:122] "RemoveContainer" containerID="a9aaa7f49ae7485a672e5834a34e890676c870b87e2aa73aec51243ae5bd963a" Nov 03 04:00:17.988032 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:17.988011 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm_odh-ai-gateway-infra(d52910ab-2950-47db-89f3-b83034339c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" podUID="d52910ab-2950-47db-89f3-b83034339c65" Nov 03 04:00:17.990042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.990018 5275 generic.go:356] "Generic (PLEG): container finished" podID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerID="1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257" exitCode=60 Nov 03 04:00:17.990219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.990201 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerDied","Data":"1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257"} Nov 03 04:00:17.990600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:17.990569 5275 scope.go:122] "RemoveContainer" containerID="1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257" Nov 03 04:00:17.990879 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:17.990823 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz_odh-ai-gateway-infra(1a743730-708a-4b42-8ce8-a7ee65693f1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" Nov 03 04:00:18.156251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.156213 5275 scope.go:122] "RemoveContainer" containerID="ef406bdd28842d6e0f7198734869cc3c82623e727a473d0ccfd40f432073b142" Nov 03 04:00:18.223963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.223929 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:18.306831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.306793 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\" (UniqueName: \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\") pod \"8e26e335-dd3c-420f-be37-90fa55b62d31\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " Nov 03 04:00:18.318220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.318192 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\") pod \"8e26e335-dd3c-420f-be37-90fa55b62d31\" (UID: \"8e26e335-dd3c-420f-be37-90fa55b62d31\") " Nov 03 04:00:18.325894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.325840 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca" pod "8e26e335-dd3c-420f-be37-90fa55b62d31" (UID: "8e26e335-dd3c-420f-be37-90fa55b62d31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:18.351901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.351493 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm" pod "8e26e335-dd3c-420f-be37-90fa55b62d31" (UID: "8e26e335-dd3c-420f-be37-90fa55b62d31"). InnerVolumeSpecName "kube-api-access-6ftmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:18.371260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.371236 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:18.375398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.375379 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:18.381241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.381217 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:18.419276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419246 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\") pod \"7363712c-e020-4595-81be-6e008381c837\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " Nov 03 04:00:18.419482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419356 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\" (UniqueName: \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\") pod \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " Nov 03 04:00:18.419482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\") pod \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\" (UID: \"8d380252-e591-4f85-8b8d-5d657fb6b8fb\") " Nov 03 04:00:18.419634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419513 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\" (UniqueName: \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\") pod \"094e43d5-02a6-4df5-ba0c-c6db69a02418\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " Nov 03 04:00:18.419634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419548 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\") pod \"094e43d5-02a6-4df5-ba0c-c6db69a02418\" (UID: \"094e43d5-02a6-4df5-ba0c-c6db69a02418\") " Nov 03 04:00:18.419758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419633 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\" (UniqueName: \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\") pod \"7363712c-e020-4595-81be-6e008381c837\" (UID: \"7363712c-e020-4595-81be-6e008381c837\") " Nov 03 04:00:18.419758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419672 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca" pod "7363712c-e020-4595-81be-6e008381c837" (UID: "7363712c-e020-4595-81be-6e008381c837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.419921 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca" pod "8d380252-e591-4f85-8b8d-5d657fb6b8fb" (UID: "8d380252-e591-4f85-8b8d-5d657fb6b8fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420061 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca" pod "094e43d5-02a6-4df5-ba0c-c6db69a02418" (UID: "094e43d5-02a6-4df5-ba0c-c6db69a02418"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420484 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d380252-e591-4f85-8b8d-5d657fb6b8fb-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420508 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/094e43d5-02a6-4df5-ba0c-c6db69a02418-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420526 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ftmm\" (UniqueName: \"kubernetes.io/projected/8e26e335-dd3c-420f-be37-90fa55b62d31-kube-api-access-6ftmm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420542 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7363712c-e020-4595-81be-6e008381c837-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.420582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.420556 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e26e335-dd3c-420f-be37-90fa55b62d31-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.449945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.449903 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g" pod "7363712c-e020-4595-81be-6e008381c837" (UID: "7363712c-e020-4595-81be-6e008381c837"). InnerVolumeSpecName "kube-api-access-vb54g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:18.450156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.449962 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx" pod "8d380252-e591-4f85-8b8d-5d657fb6b8fb" (UID: "8d380252-e591-4f85-8b8d-5d657fb6b8fb"). InnerVolumeSpecName "kube-api-access-m8vpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:18.450156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.449956 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695" pod "094e43d5-02a6-4df5-ba0c-c6db69a02418" (UID: "094e43d5-02a6-4df5-ba0c-c6db69a02418"). InnerVolumeSpecName "kube-api-access-2b695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:18.521814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.521774 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b695\" (UniqueName: \"kubernetes.io/projected/094e43d5-02a6-4df5-ba0c-c6db69a02418-kube-api-access-2b695\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.521814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.521809 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb54g\" (UniqueName: \"kubernetes.io/projected/7363712c-e020-4595-81be-6e008381c837-kube-api-access-vb54g\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.521814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.521822 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vpx\" (UniqueName: \"kubernetes.io/projected/8d380252-e591-4f85-8b8d-5d657fb6b8fb-kube-api-access-m8vpx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:18.591164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:18.591127 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:19.029453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.029421 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerStarted","Data":"3688455273fe87c3e39468baf3d47baa6182ad9166997f17b8b5c322c0cb8af7"} Nov 03 04:00:19.030905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.030877 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" event={"ID":"094e43d5-02a6-4df5-ba0c-c6db69a02418","Type":"ContainerDied","Data":"e5d9d519387136e50cf5ac8f915356a16282270d9ca13c3274a0850f333e53cd"} Nov 03 04:00:19.031020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.030945 5275 scope.go:122] "RemoveContainer" containerID="566501e206603dd53f976a35c4d67c67e2d4f7703ed20e5cdcaef7eea1c1415a" Nov 03 04:00:19.031020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.030888 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk" Nov 03 04:00:19.033182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.033157 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerStarted","Data":"e9eec8c4a4a16a6e2710583b52f7454fdd4fc0467d996f01cd449fd8c5800cd9"} Nov 03 04:00:19.035133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.035118 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" Nov 03 04:00:19.035234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.035134 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x" event={"ID":"8d380252-e591-4f85-8b8d-5d657fb6b8fb","Type":"ContainerDied","Data":"a77c836b12c294c3532395b70132771d65ff384db9a64baccc8f98971e681696"} Nov 03 04:00:19.036660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.036636 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" event={"ID":"7363712c-e020-4595-81be-6e008381c837","Type":"ContainerDied","Data":"86e31c64242ed7445c6bb8f82e689dc03a13a147706502637723a3844ee6fce5"} Nov 03 04:00:19.036762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.036648 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm" Nov 03 04:00:19.038187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.038168 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" Nov 03 04:00:19.038288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.038198 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz" event={"ID":"8e26e335-dd3c-420f-be37-90fa55b62d31","Type":"ContainerDied","Data":"58fa7c40e58845725ec95dab63f7003f2a4d8eb4a2a14c6de2bbb9eab2f0f646"} Nov 03 04:00:19.043606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.043586 5275 scope.go:122] "RemoveContainer" containerID="2317f2d897f425421d80a3cb3df29bb63a17b1d76325c29fca6fb3aac23e9cd7" Nov 03 04:00:19.044357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.044331 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerStarted","Data":"0b5c7faea28889687d51f0f09c89b5fc3e1ba1f5d5bc460a64f2bba3873cb990"} Nov 03 04:00:19.046397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.046374 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerStarted","Data":"ab44095feba6e5c835a7a2d5e7d064f14c7300b7f532b5f7c55492947e135202"} Nov 03 04:00:19.056811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.056785 5275 scope.go:122] "RemoveContainer" containerID="800c35fda0aabfedd76ba3cfe7eb73397e68391af47cf2d3667b99b177e07e51" Nov 03 04:00:19.069582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.069558 5275 scope.go:122] "RemoveContainer" containerID="26741d01db225951c9a359df738dbcfd04295c4c15042d78b3aa85234aec9cfd" Nov 03 04:00:19.293774 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.293739 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:19.297193 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.297160 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894640-d5fzk"] Nov 03 04:00:19.466234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.466163 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:19.534540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.534508 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\" (UniqueName: \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\") pod \"1a743730-708a-4b42-8ce8-a7ee65693f1e\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " Nov 03 04:00:19.534733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.534559 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\") pod \"1a743730-708a-4b42-8ce8-a7ee65693f1e\" (UID: \"1a743730-708a-4b42-8ce8-a7ee65693f1e\") " Nov 03 04:00:19.534984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.534961 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca" pod "1a743730-708a-4b42-8ce8-a7ee65693f1e" (UID: "1a743730-708a-4b42-8ce8-a7ee65693f1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:19.535114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.535099 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a743730-708a-4b42-8ce8-a7ee65693f1e-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:19.537292 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.537260 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck" pod "1a743730-708a-4b42-8ce8-a7ee65693f1e" (UID: "1a743730-708a-4b42-8ce8-a7ee65693f1e"). InnerVolumeSpecName "kube-api-access-frmck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:19.610696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.610650 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:19.642710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.639473 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frmck\" (UniqueName: \"kubernetes.io/projected/1a743730-708a-4b42-8ce8-a7ee65693f1e-kube-api-access-frmck\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:19.770892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.761560 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:19.770892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.768281 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894640-slj4x"] Nov 03 04:00:19.804491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.804464 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:19.943449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.943410 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\" (UniqueName: \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\") pod \"d52910ab-2950-47db-89f3-b83034339c65\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " Nov 03 04:00:19.943654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.943467 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\") pod \"d52910ab-2950-47db-89f3-b83034339c65\" (UID: \"d52910ab-2950-47db-89f3-b83034339c65\") " Nov 03 04:00:19.944673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.944605 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca" pod "d52910ab-2950-47db-89f3-b83034339c65" (UID: "d52910ab-2950-47db-89f3-b83034339c65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:19.951349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.945213 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d52910ab-2950-47db-89f3-b83034339c65-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:19.955138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.955094 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db" pod "d52910ab-2950-47db-89f3-b83034339c65" (UID: "d52910ab-2950-47db-89f3-b83034339c65"). InnerVolumeSpecName "kube-api-access-fg7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:19.989312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.989231 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:19.993349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:19.993315 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894640-ntrzz"] Nov 03 04:00:20.046274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.046239 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7db\" (UniqueName: \"kubernetes.io/projected/d52910ab-2950-47db-89f3-b83034339c65-kube-api-access-fg7db\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:20.055921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.055892 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" Nov 03 04:00:20.056084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.055890 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm" event={"ID":"d52910ab-2950-47db-89f3-b83034339c65","Type":"ContainerDied","Data":"15c8e3b37262b7a3b806291514acad98ff0226c4cc8ffdd80266b83a6a5a6fa3"} Nov 03 04:00:20.056084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.056004 5275 scope.go:122] "RemoveContainer" containerID="a9aaa7f49ae7485a672e5834a34e890676c870b87e2aa73aec51243ae5bd963a" Nov 03 04:00:20.057471 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.057454 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" Nov 03 04:00:20.057572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.057459 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz" event={"ID":"1a743730-708a-4b42-8ce8-a7ee65693f1e","Type":"ContainerDied","Data":"9de55a605019a0f41b2928403f66012b659180f37617c0f33281b749fe13f039"} Nov 03 04:00:20.059622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.059599 5275 generic.go:356] "Generic (PLEG): container finished" podID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerID="0b5c7faea28889687d51f0f09c89b5fc3e1ba1f5d5bc460a64f2bba3873cb990" exitCode=60 Nov 03 04:00:20.059993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.059964 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerDied","Data":"0b5c7faea28889687d51f0f09c89b5fc3e1ba1f5d5bc460a64f2bba3873cb990"} Nov 03 04:00:20.060408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.060390 5275 scope.go:122] "RemoveContainer" containerID="0b5c7faea28889687d51f0f09c89b5fc3e1ba1f5d5bc460a64f2bba3873cb990" Nov 03 04:00:20.060669 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:20.060649 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw_odh-ai-gateway-infra(3a0d19a9-8199-4def-b079-9f66b2c82c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" Nov 03 04:00:20.061718 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.061698 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerID="ab44095feba6e5c835a7a2d5e7d064f14c7300b7f532b5f7c55492947e135202" exitCode=60 Nov 03 04:00:20.061816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.061771 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerDied","Data":"ab44095feba6e5c835a7a2d5e7d064f14c7300b7f532b5f7c55492947e135202"} Nov 03 04:00:20.062187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.062163 5275 scope.go:122] "RemoveContainer" containerID="ab44095feba6e5c835a7a2d5e7d064f14c7300b7f532b5f7c55492947e135202" Nov 03 04:00:20.062406 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:20.062387 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw_odh-ai-gateway-infra(8d784e14-958f-48c8-b4e9-4efeaf79855f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" Nov 03 04:00:20.064656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.064633 5275 generic.go:356] "Generic (PLEG): container finished" podID="faf74253-7445-427d-a573-3fd58a0690d3" containerID="3688455273fe87c3e39468baf3d47baa6182ad9166997f17b8b5c322c0cb8af7" exitCode=60 Nov 03 04:00:20.064734 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.064693 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerDied","Data":"3688455273fe87c3e39468baf3d47baa6182ad9166997f17b8b5c322c0cb8af7"} Nov 03 04:00:20.065067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.065049 5275 scope.go:122] "RemoveContainer" containerID="3688455273fe87c3e39468baf3d47baa6182ad9166997f17b8b5c322c0cb8af7" Nov 03 04:00:20.065310 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:20.065287 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw_odh-ai-gateway-infra(faf74253-7445-427d-a573-3fd58a0690d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" podUID="faf74253-7445-427d-a573-3fd58a0690d3" Nov 03 04:00:20.068210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.068189 5275 generic.go:356] "Generic (PLEG): container finished" podID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerID="e9eec8c4a4a16a6e2710583b52f7454fdd4fc0467d996f01cd449fd8c5800cd9" exitCode=60 Nov 03 04:00:20.068329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.068254 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerDied","Data":"e9eec8c4a4a16a6e2710583b52f7454fdd4fc0467d996f01cd449fd8c5800cd9"} Nov 03 04:00:20.068665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.068643 5275 scope.go:122] "RemoveContainer" containerID="e9eec8c4a4a16a6e2710583b52f7454fdd4fc0467d996f01cd449fd8c5800cd9" Nov 03 04:00:20.069213 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:20.069190 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj_odh-ai-gateway-infra(657ed413-18db-474d-8695-79fe8ea9bbf3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" Nov 03 04:00:20.069277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.069205 5275 scope.go:122] "RemoveContainer" containerID="1c7348c116bce98b2e0c2324ad95185565a5230f4c5a02df0479178467acd257" Nov 03 04:00:20.080037 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.080019 5275 scope.go:122] "RemoveContainer" containerID="fd4fa35d68e9ee369f4b5dcf091dc4ffd22fc379054085dc784d4853a7b33a63" Nov 03 04:00:20.096611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.096588 5275 scope.go:122] "RemoveContainer" containerID="5e5112f6026f93d57416b731a91831b4c0725f5f7ea3b29a02c2cf05f03974d7" Nov 03 04:00:20.101481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.101452 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:20.107013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.106988 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894640-lqkwm"] Nov 03 04:00:20.112607 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.112575 5275 scope.go:122] "RemoveContainer" containerID="80b905a9556cd16dcd76bd2803b4fc78d05befb3d00b52fbf93aa2b1268a8307" Nov 03 04:00:20.128157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.128122 5275 scope.go:122] "RemoveContainer" containerID="d1fc602ccc26004568d44e8c82100af60a8624b75feb095b4b2d58e0ed218dcf" Nov 03 04:00:20.395081 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.395035 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:20.399101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.399067 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894640-smgfz"] Nov 03 04:00:20.541936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.541890 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:20.656539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.656454 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:20.890074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.890040 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:20.921491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.921387 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:20.926092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.926067 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894640-sdlkm"] Nov 03 04:00:20.986754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.986723 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" path="/var/lib/kubelet/pods/094e43d5-02a6-4df5-ba0c-c6db69a02418/volumes" Nov 03 04:00:20.987197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.987182 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" path="/var/lib/kubelet/pods/1a743730-708a-4b42-8ce8-a7ee65693f1e/volumes" Nov 03 04:00:20.987544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.987530 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7363712c-e020-4595-81be-6e008381c837" path="/var/lib/kubelet/pods/7363712c-e020-4595-81be-6e008381c837/volumes" Nov 03 04:00:20.988260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.988246 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" path="/var/lib/kubelet/pods/8d380252-e591-4f85-8b8d-5d657fb6b8fb/volumes" Nov 03 04:00:20.988756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.988718 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" path="/var/lib/kubelet/pods/8e26e335-dd3c-420f-be37-90fa55b62d31/volumes" Nov 03 04:00:20.989158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:20.989144 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52910ab-2950-47db-89f3-b83034339c65" path="/var/lib/kubelet/pods/d52910ab-2950-47db-89f3-b83034339c65/volumes" Nov 03 04:00:21.233290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.233261 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:21.370362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.370330 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:21.373507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.373484 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:21.379575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.379552 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:21.450858 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.450815 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:21.464990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.464912 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\") pod \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " Nov 03 04:00:21.464990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.464977 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\") pod \"8d784e14-958f-48c8-b4e9-4efeaf79855f\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " Nov 03 04:00:21.465187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465012 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\" (UniqueName: \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\") pod \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\" (UID: \"3a0d19a9-8199-4def-b079-9f66b2c82c3e\") " Nov 03 04:00:21.465187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465054 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\" (UniqueName: \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\") pod \"657ed413-18db-474d-8695-79fe8ea9bbf3\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " Nov 03 04:00:21.465187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465109 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\") pod \"657ed413-18db-474d-8695-79fe8ea9bbf3\" (UID: \"657ed413-18db-474d-8695-79fe8ea9bbf3\") " Nov 03 04:00:21.465187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465141 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\" (UniqueName: \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\") pod \"8d784e14-958f-48c8-b4e9-4efeaf79855f\" (UID: \"8d784e14-958f-48c8-b4e9-4efeaf79855f\") " Nov 03 04:00:21.465421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465315 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca" pod "3a0d19a9-8199-4def-b079-9f66b2c82c3e" (UID: "3a0d19a9-8199-4def-b079-9f66b2c82c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:21.465491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465429 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca" pod "8d784e14-958f-48c8-b4e9-4efeaf79855f" (UID: "8d784e14-958f-48c8-b4e9-4efeaf79855f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:21.465551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465507 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca" pod "657ed413-18db-474d-8695-79fe8ea9bbf3" (UID: "657ed413-18db-474d-8695-79fe8ea9bbf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:21.465750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465728 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d19a9-8199-4def-b079-9f66b2c82c3e-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.465814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465758 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d784e14-958f-48c8-b4e9-4efeaf79855f-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.465814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.465772 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657ed413-18db-474d-8695-79fe8ea9bbf3-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.467329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.467309 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p" pod "3a0d19a9-8199-4def-b079-9f66b2c82c3e" (UID: "3a0d19a9-8199-4def-b079-9f66b2c82c3e"). InnerVolumeSpecName "kube-api-access-x568p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:21.467429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.467364 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq" pod "657ed413-18db-474d-8695-79fe8ea9bbf3" (UID: "657ed413-18db-474d-8695-79fe8ea9bbf3"). InnerVolumeSpecName "kube-api-access-k8vrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:21.467469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.467433 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf" pod "8d784e14-958f-48c8-b4e9-4efeaf79855f" (UID: "8d784e14-958f-48c8-b4e9-4efeaf79855f"). InnerVolumeSpecName "kube-api-access-g5wdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:21.566456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.566399 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\") pod \"faf74253-7445-427d-a573-3fd58a0690d3\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " Nov 03 04:00:21.566703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.566654 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\" (UniqueName: \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\") pod \"faf74253-7445-427d-a573-3fd58a0690d3\" (UID: \"faf74253-7445-427d-a573-3fd58a0690d3\") " Nov 03 04:00:21.567095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.567072 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x568p\" (UniqueName: \"kubernetes.io/projected/3a0d19a9-8199-4def-b079-9f66b2c82c3e-kube-api-access-x568p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.567095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.567093 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vrq\" (UniqueName: \"kubernetes.io/projected/657ed413-18db-474d-8695-79fe8ea9bbf3-kube-api-access-k8vrq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.567266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.567103 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wdf\" (UniqueName: \"kubernetes.io/projected/8d784e14-958f-48c8-b4e9-4efeaf79855f-kube-api-access-g5wdf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.567266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.567177 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca" pod "faf74253-7445-427d-a573-3fd58a0690d3" (UID: "faf74253-7445-427d-a573-3fd58a0690d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:21.569387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.569359 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z" pod "faf74253-7445-427d-a573-3fd58a0690d3" (UID: "faf74253-7445-427d-a573-3fd58a0690d3"). InnerVolumeSpecName "kube-api-access-btq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:21.668700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.668662 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf74253-7445-427d-a573-3fd58a0690d3-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:21.668700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:21.668694 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btq6z\" (UniqueName: \"kubernetes.io/projected/faf74253-7445-427d-a573-3fd58a0690d3-kube-api-access-btq6z\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:22.097911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.097876 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" Nov 03 04:00:22.098130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.097877 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw" event={"ID":"3a0d19a9-8199-4def-b079-9f66b2c82c3e","Type":"ContainerDied","Data":"fe33060cc5bace90c31d22c173ea48a364bedefc67c88bd4330809af8cdefbd0"} Nov 03 04:00:22.098130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.098018 5275 scope.go:122] "RemoveContainer" containerID="0b5c7faea28889687d51f0f09c89b5fc3e1ba1f5d5bc460a64f2bba3873cb990" Nov 03 04:00:22.099433 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.099412 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" Nov 03 04:00:22.099521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.099411 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw" event={"ID":"8d784e14-958f-48c8-b4e9-4efeaf79855f","Type":"ContainerDied","Data":"fa820dd7e0b16b5a93497c1c7a8e764c92d1cd8ef6d93c7b721d5eee24feb448"} Nov 03 04:00:22.101053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.101027 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" event={"ID":"faf74253-7445-427d-a573-3fd58a0690d3","Type":"ContainerDied","Data":"95f107b5e30a92f2eeaedc3f52fb6e45afd8ca37a3abf377c0134c5b98dd8331"} Nov 03 04:00:22.101198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.101023 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw" Nov 03 04:00:22.102675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.102658 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" Nov 03 04:00:22.102789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.102745 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj" event={"ID":"657ed413-18db-474d-8695-79fe8ea9bbf3","Type":"ContainerDied","Data":"52d186e7ee9db096fba3b0abc94a8b7c0ac1cdf97b9335807a47f72ee3317006"} Nov 03 04:00:22.116077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.113475 5275 scope.go:122] "RemoveContainer" containerID="ab44095feba6e5c835a7a2d5e7d064f14c7300b7f532b5f7c55492947e135202" Nov 03 04:00:22.125021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.124998 5275 scope.go:122] "RemoveContainer" containerID="3688455273fe87c3e39468baf3d47baa6182ad9166997f17b8b5c322c0cb8af7" Nov 03 04:00:22.134420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.134402 5275 scope.go:122] "RemoveContainer" containerID="e9eec8c4a4a16a6e2710583b52f7454fdd4fc0467d996f01cd449fd8c5800cd9" Nov 03 04:00:22.262886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.262807 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:22.266290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.266254 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894640-8t7nw"] Nov 03 04:00:22.378916 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.378819 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:22.382996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.382966 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894640-q6tzw"] Nov 03 04:00:22.491749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.491612 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:22.495412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.495384 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894640-8knnw"] Nov 03 04:00:22.603038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.603006 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:22.607345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.607310 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894640-fmlmj"] Nov 03 04:00:22.985398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.985366 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" path="/var/lib/kubelet/pods/3a0d19a9-8199-4def-b079-9f66b2c82c3e/volumes" Nov 03 04:00:22.985773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.985760 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" path="/var/lib/kubelet/pods/657ed413-18db-474d-8695-79fe8ea9bbf3/volumes" Nov 03 04:00:22.986143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.986130 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" path="/var/lib/kubelet/pods/8d784e14-958f-48c8-b4e9-4efeaf79855f/volumes" Nov 03 04:00:22.986802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:00:22.986786 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf74253-7445-427d-a573-3fd58a0690d3" path="/var/lib/kubelet/pods/faf74253-7445-427d-a573-3fd58a0690d3/volumes" Nov 03 04:00:32.333196 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:00:32.333157 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:01:02.330311 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:01:02.330273 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:01:24.652511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.652477 5275 generic.go:356] "Generic (PLEG): container finished" podID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerID="6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c" exitCode=0 Nov 03 04:01:24.652961 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.652545 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerDied","Data":"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c"} Nov 03 04:01:24.653020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.652975 5275 scope.go:122] "RemoveContainer" containerID="6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c" Nov 03 04:01:24.654521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.654493 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:01:24.654593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.654564 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:01:24.654634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:24.654597 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:01:25.682795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:25.682755 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerStarted","Data":"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87"} Nov 03 04:01:25.889178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:25.889136 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:01:25.890427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:25.890398 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 04:01:25.890582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:25.890460 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 04:01:26.581879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:26.581810 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:01:26.889055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:26.888967 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 04:01:26.889444 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:26.889048 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 04:01:27.889385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:27.889350 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 04:01:27.889807 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:27.889431 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 04:01:28.889342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:28.889306 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 04:01:28.889548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:28.889364 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 04:01:29.889285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:29.889242 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 04:01:29.889502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:29.889319 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 04:01:30.893663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:30.893627 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:01:30.896301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:30.896277 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:01:32.333861 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:01:32.333807 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:01:50.011258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:50.011226 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:01:50.022771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:01:50.022744 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:02:02.333511 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:02:02.333475 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:02:10.142444 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.142403 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7113-3b9fa2"] Nov 03 04:02:10.142991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.142976 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.142992 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143004 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143008 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143016 5275 cpu_manager.go:414] "containerMap: removing container" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143021 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143028 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143033 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143040 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.143044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143046 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143056 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143061 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143070 5275 cpu_manager.go:414] "containerMap: removing container" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143074 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143079 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143083 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143088 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143093 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143099 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143104 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143112 5275 cpu_manager.go:414] "containerMap: removing container" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143117 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143124 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143129 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143135 5275 cpu_manager.go:414] "containerMap: removing container" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143139 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143147 5275 cpu_manager.go:414] "containerMap: removing container" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143151 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143158 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143162 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143174 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143178 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143186 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143191 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143197 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143202 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143210 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143214 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143227 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143231 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143238 5275 cpu_manager.go:414] "containerMap: removing container" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143242 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143248 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143251 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143257 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143260 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143269 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143273 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143281 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.143335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143285 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143361 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143368 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143374 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143380 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143386 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143392 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143397 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143402 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143406 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143412 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143416 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143420 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143426 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143430 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143436 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143440 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143445 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143450 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143456 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143461 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7363712c-e020-4595-81be-6e008381c837" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143465 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143471 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143478 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143483 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a743730-708a-4b42-8ce8-a7ee65693f1e" containerName="cleanup" Nov 03 04:02:10.144773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.143488 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:10.145983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.145679 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:10.199160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.199130 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr9t9\"" Nov 03 04:02:10.202449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.202414 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-3b9fa2"] Nov 03 04:02:10.315671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.315638 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzl9\" (UniqueName: \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\") pod \"test-curl-7113-3b9fa2\" (UID: \"f2cba08a-0241-46a1-bdca-412f7b07dbad\") " pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:10.416773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.416676 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzl9\" (UniqueName: \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\") pod \"test-curl-7113-3b9fa2\" (UID: \"f2cba08a-0241-46a1-bdca-412f7b07dbad\") " pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:10.478768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.478730 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzl9\" (UniqueName: \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\") pod \"test-curl-7113-3b9fa2\" (UID: \"f2cba08a-0241-46a1-bdca-412f7b07dbad\") " pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:10.759397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:10.759307 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:11.135924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.135896 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-3b9fa2"] Nov 03 04:02:11.137360 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:11.137333 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cba08a_0241_46a1_bdca_412f7b07dbad.slice/crio-e5d100853aa7c1b2692c73bc590e517555901a324bb3364693193cde31d735e4 WatchSource:0}: Error finding container e5d100853aa7c1b2692c73bc590e517555901a324bb3364693193cde31d735e4: Status 404 returned error can't find the container with id e5d100853aa7c1b2692c73bc590e517555901a324bb3364693193cde31d735e4 Nov 03 04:02:11.613338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.613295 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb"] Nov 03 04:02:11.613975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.613953 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:11.613975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.613975 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.613992 5275 cpu_manager.go:414] "containerMap: removing container" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.613997 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf74253-7445-427d-a573-3fd58a0690d3" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614009 5275 cpu_manager.go:414] "containerMap: removing container" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614014 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e43d5-02a6-4df5-ba0c-c6db69a02418" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614047 5275 cpu_manager.go:414] "containerMap: removing container" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614051 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657ed413-18db-474d-8695-79fe8ea9bbf3" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614080 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:11.614143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614085 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:11.614608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614175 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d380252-e591-4f85-8b8d-5d657fb6b8fb" containerName="cleanup" Nov 03 04:02:11.614608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614190 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0d19a9-8199-4def-b079-9f66b2c82c3e" containerName="cleanup" Nov 03 04:02:11.614608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614198 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e26e335-dd3c-420f-be37-90fa55b62d31" containerName="cleanup" Nov 03 04:02:11.614608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614207 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52910ab-2950-47db-89f3-b83034339c65" containerName="cleanup" Nov 03 04:02:11.614608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.614215 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d784e14-958f-48c8-b4e9-4efeaf79855f" containerName="cleanup" Nov 03 04:02:11.618176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.616628 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.674669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.673490 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:02:11.674669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.673602 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:02:11.674669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.674018 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:02:11.674669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.674404 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:02:11.675073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.675046 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb"] Nov 03 04:02:11.688675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.688625 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw"] Nov 03 04:02:11.698055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.698024 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.736693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.736658 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgsn\" (UniqueName: \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.736882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.736859 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.736950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.736918 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.755613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.755575 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw"] Nov 03 04:02:11.838243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838203 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.838243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838250 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsn\" (UniqueName: \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.838486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838336 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.838486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838482 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.838597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838513 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.838597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.838545 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b5r\" (UniqueName: \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.839197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.839167 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.841136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.841114 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.899176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.899100 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgsn\" (UniqueName: \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\") pod \"maas-api-revoke-sub-5d5ee23b-pldwb\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.937271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.937239 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:11.939740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.939624 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b5r\" (UniqueName: \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.939740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.939704 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.939918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.939741 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.940628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.940599 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.942194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.942175 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:11.979632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.979593 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg"] Nov 03 04:02:11.983924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:11.983901 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.001286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.001149 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b5r\" (UniqueName: \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\") pod \"maas-api-revoke-sub-54e9d836-2gcmw\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:12.011243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.009427 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-3b9fa2" event={"ID":"f2cba08a-0241-46a1-bdca-412f7b07dbad","Type":"ContainerStarted","Data":"233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49"} Nov 03 04:02:12.011243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.009481 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-3b9fa2" event={"ID":"f2cba08a-0241-46a1-bdca-412f7b07dbad","Type":"ContainerStarted","Data":"e5d100853aa7c1b2692c73bc590e517555901a324bb3364693193cde31d735e4"} Nov 03 04:02:12.042530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.042495 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:12.047318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.045215 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg"] Nov 03 04:02:12.142921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.142401 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.142921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.142483 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.142921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.142634 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmh6k\" (UniqueName: \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.215367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.215274 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7113-3b9fa2" podStartSLOduration=2.215259503 podStartE2EDuration="2.215259503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:12.213474059 +0000 UTC m=+3623.366162481" watchObservedRunningTime="2026-11-03 04:02:12.215259503 +0000 UTC m=+3623.367947923" Nov 03 04:02:12.244165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.244129 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmh6k\" (UniqueName: \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.244351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.244268 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.244351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.244302 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.245053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.245032 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.247096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.247070 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.258625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.258590 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:02:12.263444 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.263412 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.341494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.335778 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-03da9e-openshift-default-dockercfg-9gdmc\"" Nov 03 04:02:12.346195 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346151 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346433 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346340 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346437 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdw9\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346487 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346523 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346565 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346614 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346667 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.346703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.346691 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.356082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.356050 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmh6k\" (UniqueName: \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\") pod \"maas-api-revoke-sub-ac3de38c-949pg\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:12.372420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.366978 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:02:12.372420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.369676 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb"] Nov 03 04:02:12.447775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.447741 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.447945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.447874 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.447962 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdw9\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.447996 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448032 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448076 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448142 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448204 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448401 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448379 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448554 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448535 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448561 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448570 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448617 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.448957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.448829 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.451737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.451712 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.451990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.451967 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.476044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.476017 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw"] Nov 03 04:02:12.550341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.550303 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.550747 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.550726 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdw9\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\") pod \"e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.552062 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:12.552036 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e1224e_62ef_4cf5_9882_e3b9f2cd5abe.slice/crio-97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e WatchSource:0}: Error finding container 97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e: Status 404 returned error can't find the container with id 97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e Nov 03 04:02:12.578393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.578370 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:12.606130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:12.606093 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:13.010138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.010110 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg"] Nov 03 04:02:13.013814 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:13.012064 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be6ccaf_95fe_4749_8b75_84f21f9084fd.slice/crio-fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9 WatchSource:0}: Error finding container fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9: Status 404 returned error can't find the container with id fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9 Nov 03 04:02:13.017237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.017211 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:02:13.017465 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:13.017439 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5db47a_df8b_4df0_8d97_112092993e04.slice/crio-921e0ce170abba32d20b3d2c514214e88a0731bd865493cee266324d74848620 WatchSource:0}: Error finding container 921e0ce170abba32d20b3d2c514214e88a0731bd865493cee266324d74848620: Status 404 returned error can't find the container with id 921e0ce170abba32d20b3d2c514214e88a0731bd865493cee266324d74848620 Nov 03 04:02:13.023030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.023001 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" event={"ID":"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe","Type":"ContainerStarted","Data":"acbfc0703f6166ee6aaa9be15bbf79a6d05186d06667ce8652ac5fe326614481"} Nov 03 04:02:13.023147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.023093 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" event={"ID":"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe","Type":"ContainerStarted","Data":"97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e"} Nov 03 04:02:13.024751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.024726 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" event={"ID":"4daa4c56-06bc-4a71-93dd-17b29c3af2b7","Type":"ContainerStarted","Data":"eb6b23a22902c5a0e8b4e2e2e4c22ab9d40fdace0b6789b0c4f59b20bd6fba44"} Nov 03 04:02:13.024884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.024801 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" event={"ID":"4daa4c56-06bc-4a71-93dd-17b29c3af2b7","Type":"ContainerStarted","Data":"3dcf03a7d88888b7b56dc329ed87324324f1d8d1a116e421cc72a27841fb0cd3"} Nov 03 04:02:13.026145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.026117 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:13.026219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.026179 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:13.026219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.026207 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:13.140522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.140469 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" podStartSLOduration=2.140455157 podStartE2EDuration="2.140455157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:13.137915011 +0000 UTC m=+3624.290603433" watchObservedRunningTime="2026-11-03 04:02:13.140455157 +0000 UTC m=+3624.293143578" Nov 03 04:02:13.255306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.255231 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" podStartSLOduration=2.255213405 podStartE2EDuration="2.255213405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:13.252638096 +0000 UTC m=+3624.405326519" watchObservedRunningTime="2026-11-03 04:02:13.255213405 +0000 UTC m=+3624.407901827" Nov 03 04:02:13.664280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.664248 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7113-3b9fa2"] Nov 03 04:02:13.669887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.669840 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7113-3b9fa2"] Nov 03 04:02:13.946566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.946532 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77"] Nov 03 04:02:13.947174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.947149 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" containerName="test-curl-7113-3b9fa2" Nov 03 04:02:13.947174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.947166 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" containerName="test-curl-7113-3b9fa2" Nov 03 04:02:13.947312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.947301 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" containerName="test-curl-7113-3b9fa2" Nov 03 04:02:13.950684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:13.950662 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.008309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.008274 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77"] Nov 03 04:02:14.040962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.040907 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerStarted","Data":"2b32c04d53f6f345aa3b537021856a40cd06320e7be7c3cabbd7e60a949eb192"} Nov 03 04:02:14.041448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.041025 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerStarted","Data":"921e0ce170abba32d20b3d2c514214e88a0731bd865493cee266324d74848620"} Nov 03 04:02:14.052491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.052455 5275 generic.go:356] "Generic (PLEG): container finished" podID="89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" containerID="acbfc0703f6166ee6aaa9be15bbf79a6d05186d06667ce8652ac5fe326614481" exitCode=0 Nov 03 04:02:14.052901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.052839 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" event={"ID":"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe","Type":"ContainerDied","Data":"acbfc0703f6166ee6aaa9be15bbf79a6d05186d06667ce8652ac5fe326614481"} Nov 03 04:02:14.055749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.055720 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" event={"ID":"9be6ccaf-95fe-4749-8b75-84f21f9084fd","Type":"ContainerStarted","Data":"ef92495e503ac9182ea8d94193dcaaa9731249d268e5009a2f379d304bd75595"} Nov 03 04:02:14.055892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.055782 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" event={"ID":"9be6ccaf-95fe-4749-8b75-84f21f9084fd","Type":"ContainerStarted","Data":"fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9"} Nov 03 04:02:14.061495 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.061465 5275 generic.go:356] "Generic (PLEG): container finished" podID="4daa4c56-06bc-4a71-93dd-17b29c3af2b7" containerID="eb6b23a22902c5a0e8b4e2e2e4c22ab9d40fdace0b6789b0c4f59b20bd6fba44" exitCode=0 Nov 03 04:02:14.061750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.061730 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7113-3b9fa2" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" containerName="test-curl-7113-3b9fa2" containerID="cri-o://233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49" gracePeriod=2 Nov 03 04:02:14.062143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.062116 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" event={"ID":"4daa4c56-06bc-4a71-93dd-17b29c3af2b7","Type":"ContainerDied","Data":"eb6b23a22902c5a0e8b4e2e2e4c22ab9d40fdace0b6789b0c4f59b20bd6fba44"} Nov 03 04:02:14.076541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.076504 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.076735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.076598 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.076735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.076662 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9755g\" (UniqueName: \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.178205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.178166 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.178396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.178276 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.178396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.178338 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9755g\" (UniqueName: \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.178994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.178969 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.182270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.182242 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.183981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.183938 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" podStartSLOduration=3.183925768 podStartE2EDuration="3.183925768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:14.181153829 +0000 UTC m=+3625.333842251" watchObservedRunningTime="2026-11-03 04:02:14.183925768 +0000 UTC m=+3625.336614188" Nov 03 04:02:14.239672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.239636 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9755g\" (UniqueName: \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\") pod \"maas-api-revoke-sub-9c8e545d-z6x77\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.265789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.265752 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:14.534570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.534516 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podStartSLOduration=2.534500905 podStartE2EDuration="2.534500905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:14.532095807 +0000 UTC m=+3625.684784229" watchObservedRunningTime="2026-11-03 04:02:14.534500905 +0000 UTC m=+3625.687189326" Nov 03 04:02:14.579592 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.579546 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:14.582801 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:02:14.582751 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-3b9fa2\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:14.586560 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.586528 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:14.653104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:14.649809 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77"] Nov 03 04:02:14.674069 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:14.674038 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d687d4_b864_4e4f_913f_09d1a7ca7e32.slice/crio-4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7 WatchSource:0}: Error finding container 4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7: Status 404 returned error can't find the container with id 4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7 Nov 03 04:02:14.757080 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:02:14.757034 5275 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-3b9fa2\" is forbidden: User \"system:node:ip-10-0-136-7.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-7.us-west-1.compute.internal' and this object" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:15.070387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.070351 5275 generic.go:356] "Generic (PLEG): container finished" podID="9be6ccaf-95fe-4749-8b75-84f21f9084fd" containerID="ef92495e503ac9182ea8d94193dcaaa9731249d268e5009a2f379d304bd75595" exitCode=0 Nov 03 04:02:15.071132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.070432 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" event={"ID":"9be6ccaf-95fe-4749-8b75-84f21f9084fd","Type":"ContainerDied","Data":"ef92495e503ac9182ea8d94193dcaaa9731249d268e5009a2f379d304bd75595"} Nov 03 04:02:15.072518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.072489 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" event={"ID":"a6d687d4-b864-4e4f-913f-09d1a7ca7e32","Type":"ContainerStarted","Data":"3b1a8aaf596f381dc69f9b5ab055657b717af1e639c8a1a8d5be845a4e254d4e"} Nov 03 04:02:15.073382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.073358 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:15.073504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.073398 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" event={"ID":"a6d687d4-b864-4e4f-913f-09d1a7ca7e32","Type":"ContainerStarted","Data":"4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7"} Nov 03 04:02:15.074103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.073719 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:02:15.246234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.246200 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:15.302185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.302130 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" podStartSLOduration=2.302111062 podStartE2EDuration="2.302111062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:15.300251897 +0000 UTC m=+3626.452940318" watchObservedRunningTime="2026-11-03 04:02:15.302111062 +0000 UTC m=+3626.454799483" Nov 03 04:02:15.308340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.308307 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:15.394159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394062 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\") pod \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " Nov 03 04:02:15.394349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394215 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\") pod \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " Nov 03 04:02:15.394349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394250 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\") pod \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " Nov 03 04:02:15.394349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394326 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\" (UniqueName: \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\") pod \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\" (UID: \"4daa4c56-06bc-4a71-93dd-17b29c3af2b7\") " Nov 03 04:02:15.394515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394352 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\" (UniqueName: \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\") pod \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " Nov 03 04:02:15.394515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394379 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\") pod \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\" (UID: \"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe\") " Nov 03 04:02:15.394515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394487 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca" pod "4daa4c56-06bc-4a71-93dd-17b29c3af2b7" (UID: "4daa4c56-06bc-4a71-93dd-17b29c3af2b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:15.394703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394670 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca" pod "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" (UID: "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:15.395013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.394989 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:15.395113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.395017 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:15.396988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.396962 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca" pod "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" (UID: "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:15.397099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.397084 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca" pod "4daa4c56-06bc-4a71-93dd-17b29c3af2b7" (UID: "4daa4c56-06bc-4a71-93dd-17b29c3af2b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:15.397285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.397265 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn" pod "4daa4c56-06bc-4a71-93dd-17b29c3af2b7" (UID: "4daa4c56-06bc-4a71-93dd-17b29c3af2b7"). InnerVolumeSpecName "kube-api-access-krgsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:15.397372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.397294 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r" pod "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" (UID: "89e1224e-62ef-4cf5-9882-e3b9f2cd5abe"). InnerVolumeSpecName "kube-api-access-99b5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:15.496605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.496570 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:15.496605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.496602 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgsn\" (UniqueName: \"kubernetes.io/projected/4daa4c56-06bc-4a71-93dd-17b29c3af2b7-kube-api-access-krgsn\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:15.496605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.496613 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b5r\" (UniqueName: \"kubernetes.io/projected/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-kube-api-access-99b5r\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:15.496964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:15.496623 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:16.085106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.085077 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" Nov 03 04:02:16.085106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.085097 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb" event={"ID":"4daa4c56-06bc-4a71-93dd-17b29c3af2b7","Type":"ContainerDied","Data":"3dcf03a7d88888b7b56dc329ed87324324f1d8d1a116e421cc72a27841fb0cd3"} Nov 03 04:02:16.086529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.085150 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dcf03a7d88888b7b56dc329ed87324324f1d8d1a116e421cc72a27841fb0cd3" Nov 03 04:02:16.087341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.087311 5275 generic.go:356] "Generic (PLEG): container finished" podID="a6d687d4-b864-4e4f-913f-09d1a7ca7e32" containerID="3b1a8aaf596f381dc69f9b5ab055657b717af1e639c8a1a8d5be845a4e254d4e" exitCode=0 Nov 03 04:02:16.087482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.087408 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" event={"ID":"a6d687d4-b864-4e4f-913f-09d1a7ca7e32","Type":"ContainerDied","Data":"3b1a8aaf596f381dc69f9b5ab055657b717af1e639c8a1a8d5be845a4e254d4e"} Nov 03 04:02:16.089951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.089925 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" Nov 03 04:02:16.090071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.089978 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw" event={"ID":"89e1224e-62ef-4cf5-9882-e3b9f2cd5abe","Type":"ContainerDied","Data":"97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e"} Nov 03 04:02:16.090071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.090023 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b1d42a1b98317ec4959e217a064a975737a5000cb23fc408937bcc49b4584e" Nov 03 04:02:16.289973 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.289940 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:16.348233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.348206 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:16.409338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.409249 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\") pod \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " Nov 03 04:02:16.409338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.409306 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\" (UniqueName: \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\") pod \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " Nov 03 04:02:16.409562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.409397 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\") pod \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\" (UID: \"9be6ccaf-95fe-4749-8b75-84f21f9084fd\") " Nov 03 04:02:16.409814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.409785 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca" pod "9be6ccaf-95fe-4749-8b75-84f21f9084fd" (UID: "9be6ccaf-95fe-4749-8b75-84f21f9084fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:16.410140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.410107 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:16.411573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.411547 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca" pod "9be6ccaf-95fe-4749-8b75-84f21f9084fd" (UID: "9be6ccaf-95fe-4749-8b75-84f21f9084fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:16.411657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.411613 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k" pod "9be6ccaf-95fe-4749-8b75-84f21f9084fd" (UID: "9be6ccaf-95fe-4749-8b75-84f21f9084fd"). InnerVolumeSpecName "kube-api-access-mmh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:16.511585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.511549 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\" (UniqueName: \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\") pod \"f2cba08a-0241-46a1-bdca-412f7b07dbad\" (UID: \"f2cba08a-0241-46a1-bdca-412f7b07dbad\") " Nov 03 04:02:16.511941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.511924 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be6ccaf-95fe-4749-8b75-84f21f9084fd-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:16.511941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.511942 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmh6k\" (UniqueName: \"kubernetes.io/projected/9be6ccaf-95fe-4749-8b75-84f21f9084fd-kube-api-access-mmh6k\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:16.513602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.513577 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9" pod "f2cba08a-0241-46a1-bdca-412f7b07dbad" (UID: "f2cba08a-0241-46a1-bdca-412f7b07dbad"). InnerVolumeSpecName "kube-api-access-krzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:16.613213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.613181 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzl9\" (UniqueName: \"kubernetes.io/projected/f2cba08a-0241-46a1-bdca-412f7b07dbad-kube-api-access-krzl9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:16.988403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:16.988359 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cba08a-0241-46a1-bdca-412f7b07dbad" path="/var/lib/kubelet/pods/f2cba08a-0241-46a1-bdca-412f7b07dbad/volumes" Nov 03 04:02:17.098803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.098769 5275 generic.go:356] "Generic (PLEG): container finished" podID="f2cba08a-0241-46a1-bdca-412f7b07dbad" containerID="233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49" exitCode=137 Nov 03 04:02:17.099255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.098829 5275 scope.go:122] "RemoveContainer" containerID="233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49" Nov 03 04:02:17.099255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.098882 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-3b9fa2" Nov 03 04:02:17.100734 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.100717 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" Nov 03 04:02:17.100880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.100742 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg" event={"ID":"9be6ccaf-95fe-4749-8b75-84f21f9084fd","Type":"ContainerDied","Data":"fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9"} Nov 03 04:02:17.100880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.100788 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe43d339f40b806098e4b1530bf7c0e9a9c72790845b4acea54b9184c94d5bd9" Nov 03 04:02:17.109585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.109563 5275 scope.go:122] "RemoveContainer" containerID="233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49" Nov 03 04:02:17.109924 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:02:17.109903 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49\": container with ID starting with 233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49 not found: ID does not exist" containerID="233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49" Nov 03 04:02:17.110045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.109944 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49"} err="failed to get container status \"233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49\": rpc error: code = NotFound desc = could not find container \"233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49\": container with ID starting with 233eab7320c45b24a56a97274202cafa17fddadcfb6175515bd101048e059b49 not found: ID does not exist" Nov 03 04:02:17.261453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.261426 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:17.422147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.422061 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\" (UniqueName: \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\") pod \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " Nov 03 04:02:17.422340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.422146 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\") pod \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " Nov 03 04:02:17.422340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.422191 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\") pod \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\" (UID: \"a6d687d4-b864-4e4f-913f-09d1a7ca7e32\") " Nov 03 04:02:17.422887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.422822 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca" pod "a6d687d4-b864-4e4f-913f-09d1a7ca7e32" (UID: "a6d687d4-b864-4e4f-913f-09d1a7ca7e32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:17.424629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.424609 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca" pod "a6d687d4-b864-4e4f-913f-09d1a7ca7e32" (UID: "a6d687d4-b864-4e4f-913f-09d1a7ca7e32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:17.424820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.424804 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g" pod "a6d687d4-b864-4e4f-913f-09d1a7ca7e32" (UID: "a6d687d4-b864-4e4f-913f-09d1a7ca7e32"). InnerVolumeSpecName "kube-api-access-9755g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:17.524092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.524058 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9755g\" (UniqueName: \"kubernetes.io/projected/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-kube-api-access-9755g\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:17.524092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.524086 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:17.524092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:17.524096 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d687d4-b864-4e4f-913f-09d1a7ca7e32-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:18.111157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:18.111119 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" Nov 03 04:02:18.111619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:18.111168 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77" event={"ID":"a6d687d4-b864-4e4f-913f-09d1a7ca7e32","Type":"ContainerDied","Data":"4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7"} Nov 03 04:02:18.111619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:18.111226 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bea6303efd9e0698712cb987a1fd37977a66b4c2337ae84dccd7ad787864ba7" Nov 03 04:02:21.246446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.246406 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:02:21.247217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247196 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a6d687d4-b864-4e4f-913f-09d1a7ca7e32" containerName="revoke-keys" Nov 03 04:02:21.247217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247218 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d687d4-b864-4e4f-913f-09d1a7ca7e32" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247243 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9be6ccaf-95fe-4749-8b75-84f21f9084fd" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247250 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be6ccaf-95fe-4749-8b75-84f21f9084fd" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247278 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4daa4c56-06bc-4a71-93dd-17b29c3af2b7" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247285 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4daa4c56-06bc-4a71-93dd-17b29c3af2b7" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247297 5275 cpu_manager.go:414] "containerMap: removing container" podUID="89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" containerName="revoke-keys" Nov 03 04:02:21.247384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247305 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" containerName="revoke-keys" Nov 03 04:02:21.247690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247467 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4daa4c56-06bc-4a71-93dd-17b29c3af2b7" containerName="revoke-keys" Nov 03 04:02:21.247690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247480 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" containerName="revoke-keys" Nov 03 04:02:21.247690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247503 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d687d4-b864-4e4f-913f-09d1a7ca7e32" containerName="revoke-keys" Nov 03 04:02:21.247690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.247515 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be6ccaf-95fe-4749-8b75-84f21f9084fd" containerName="revoke-keys" Nov 03 04:02:21.251360 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.251335 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.271146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.271116 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6gp\" (UniqueName: \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.271312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.271291 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.308187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.308153 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:02:21.384489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.384453 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-03da9e-dockercfg-bsng2\"" Nov 03 04:02:21.384671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.384518 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-03da9e\"" Nov 03 04:02:21.385639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.385608 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.385806 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.385759 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6gp\" (UniqueName: \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.386680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.386652 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.403177 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.403145 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:02:21.407689 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.407665 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.446897 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.446864 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6gp\" (UniqueName: \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\") pod \"payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.469692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.469649 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:02:21.486925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.486890 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.487082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.487023 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4w5\" (UniqueName: \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.565778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.565746 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:21.588250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.588214 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.588423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.588299 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4w5\" (UniqueName: \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.589214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.589187 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.649009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.648975 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4w5\" (UniqueName: \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\") pod \"payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.725387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.725348 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:21.945196 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:21.945169 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:02:21.945993 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:21.945968 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af9a49c_e2ac_4936_905d_fdb6d88232b4.slice/crio-6111c41639d07e12289a0867ed969361e4e2a6541cd2c09a04fa59f2d51ce295 WatchSource:0}: Error finding container 6111c41639d07e12289a0867ed969361e4e2a6541cd2c09a04fa59f2d51ce295: Status 404 returned error can't find the container with id 6111c41639d07e12289a0867ed969361e4e2a6541cd2c09a04fa59f2d51ce295 Nov 03 04:02:22.105322 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.105290 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:02:22.106293 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:22.106270 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd963bc_1b69_42ee_b90e_100e8155defb.slice/crio-f82591dc51275ffb1adc317943caf5a1df6900922df4d62c26295bbd664b222d WatchSource:0}: Error finding container f82591dc51275ffb1adc317943caf5a1df6900922df4d62c26295bbd664b222d: Status 404 returned error can't find the container with id f82591dc51275ffb1adc317943caf5a1df6900922df4d62c26295bbd664b222d Nov 03 04:02:22.147906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.147864 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" event={"ID":"4af9a49c-e2ac-4936-905d-fdb6d88232b4","Type":"ContainerStarted","Data":"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5"} Nov 03 04:02:22.148035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.147937 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:22.148035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.147961 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" event={"ID":"4af9a49c-e2ac-4936-905d-fdb6d88232b4","Type":"ContainerStarted","Data":"6111c41639d07e12289a0867ed969361e4e2a6541cd2c09a04fa59f2d51ce295"} Nov 03 04:02:22.149209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.149189 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" event={"ID":"2bd963bc-1b69-42ee-b90e-100e8155defb","Type":"ContainerStarted","Data":"f82591dc51275ffb1adc317943caf5a1df6900922df4d62c26295bbd664b222d"} Nov 03 04:02:22.264372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.263741 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" podStartSLOduration=1.26372637 podStartE2EDuration="1.26372637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:22.262895679 +0000 UTC m=+3633.415584102" watchObservedRunningTime="2026-11-03 04:02:22.26372637 +0000 UTC m=+3633.416414793" Nov 03 04:02:22.378440 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.378404 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8"] Nov 03 04:02:22.384663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.384451 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.437394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.437361 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:02:22.438640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.438616 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8"] Nov 03 04:02:22.438989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.438967 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:02:22.439243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.439225 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:02:22.439570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.439553 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:02:22.474260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.474224 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5"] Nov 03 04:02:22.479662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.479637 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.500724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.500688 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.500909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.500727 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xssjf\" (UniqueName: \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.500963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.500916 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.537040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.537001 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5"] Nov 03 04:02:22.602522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602479 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.602708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602537 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.602708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602595 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssjf\" (UniqueName: \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.602708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602678 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.602924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602734 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.602924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.602810 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7sv\" (UniqueName: \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.603373 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.603346 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.605340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.605315 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.663768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.663734 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssjf\" (UniqueName: \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\") pod \"maas-api-revoke-sub-47dbeda7-6hgq8\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.699580 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.699549 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:22.704183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.704156 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.704298 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.704279 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.704381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.704365 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7sv\" (UniqueName: \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.704913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.704892 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.706711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.706685 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.737604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.737569 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd"] Nov 03 04:02:22.742928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.742902 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.770063 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.768707 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7sv\" (UniqueName: \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\") pod \"maas-api-revoke-sub-dbd9602b-mk4b5\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.799951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.797057 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:22.804263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.801946 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd"] Nov 03 04:02:22.805739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.805701 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.805910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.805787 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgf9\" (UniqueName: \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.805992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.805933 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.907408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.907374 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.907587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.907440 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgf9\" (UniqueName: \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.907587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.907541 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.908263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.908233 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.910166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.910142 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:22.968563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:22.968532 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgf9\" (UniqueName: \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\") pod \"maas-api-revoke-sub-c2b14b58-2z4jd\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:23.060422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.060340 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:23.081570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.081545 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8"] Nov 03 04:02:23.083719 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:23.083689 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1136b2_8585_493c_8934_64e55b3389dd.slice/crio-cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4 WatchSource:0}: Error finding container cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4: Status 404 returned error can't find the container with id cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4 Nov 03 04:02:23.164887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.164559 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" event={"ID":"2bd963bc-1b69-42ee-b90e-100e8155defb","Type":"ContainerStarted","Data":"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739"} Nov 03 04:02:23.164887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.164654 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:23.167107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.167062 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" event={"ID":"5d1136b2-8585-493c-8934-64e55b3389dd","Type":"ContainerStarted","Data":"cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4"} Nov 03 04:02:23.182901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.182867 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5"] Nov 03 04:02:23.194141 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:23.194108 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80924389_892f_4e3a_95ad_3a03633f8bef.slice/crio-cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1 WatchSource:0}: Error finding container cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1: Status 404 returned error can't find the container with id cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1 Nov 03 04:02:23.281119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.281058 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" podStartSLOduration=2.281042389 podStartE2EDuration="2.281042389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:23.279620256 +0000 UTC m=+3634.432308691" watchObservedRunningTime="2026-11-03 04:02:23.281042389 +0000 UTC m=+3634.433730817" Nov 03 04:02:23.445354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:23.445322 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd"] Nov 03 04:02:23.445985 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:23.445962 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa2010a_fed1_42f7_ad27_554fb3dc6e08.slice/crio-96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e WatchSource:0}: Error finding container 96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e: Status 404 returned error can't find the container with id 96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e Nov 03 04:02:24.177769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.177732 5275 generic.go:356] "Generic (PLEG): container finished" podID="5d1136b2-8585-493c-8934-64e55b3389dd" containerID="6f3467d0a30e44f267112d797244360ba47ad261d5dd1b10c1ec89187fc19707" exitCode=0 Nov 03 04:02:24.178010 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.177823 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" event={"ID":"5d1136b2-8585-493c-8934-64e55b3389dd","Type":"ContainerDied","Data":"6f3467d0a30e44f267112d797244360ba47ad261d5dd1b10c1ec89187fc19707"} Nov 03 04:02:24.180006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.179976 5275 generic.go:356] "Generic (PLEG): container finished" podID="80924389-892f-4e3a-95ad-3a03633f8bef" containerID="7f3ec21663094561bffb2c1f5292831ad163ac1d4b13b4159e96234661d3c6d4" exitCode=0 Nov 03 04:02:24.180150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.180056 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" event={"ID":"80924389-892f-4e3a-95ad-3a03633f8bef","Type":"ContainerDied","Data":"7f3ec21663094561bffb2c1f5292831ad163ac1d4b13b4159e96234661d3c6d4"} Nov 03 04:02:24.180150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.180103 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" event={"ID":"80924389-892f-4e3a-95ad-3a03633f8bef","Type":"ContainerStarted","Data":"cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1"} Nov 03 04:02:24.181896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.181875 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" event={"ID":"2fa2010a-fed1-42f7-ad27-554fb3dc6e08","Type":"ContainerStarted","Data":"c25d9e0b3c59407602242002b940d4f90187875b64c4dd42867363c185b7c8d6"} Nov 03 04:02:24.182026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.181914 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" event={"ID":"2fa2010a-fed1-42f7-ad27-554fb3dc6e08","Type":"ContainerStarted","Data":"96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e"} Nov 03 04:02:24.408992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.408932 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" podStartSLOduration=2.408917896 podStartE2EDuration="2.408917896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:24.407896086 +0000 UTC m=+3635.560584499" watchObservedRunningTime="2026-11-03 04:02:24.408917896 +0000 UTC m=+3635.561606316" Nov 03 04:02:24.710498 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.710423 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm"] Nov 03 04:02:24.714628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.714611 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.772995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.772958 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm"] Nov 03 04:02:24.831814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.831774 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.832018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.831837 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqrq\" (UniqueName: \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.832107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.832081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.933589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.933553 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.933801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.933674 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.933801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.933716 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqrq\" (UniqueName: \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.934352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.934326 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.936201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.936179 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:24.995065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:24.994989 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqrq\" (UniqueName: \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\") pod \"maas-api-revoke-sub-f5ce77ee-j74sm\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:25.028692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.028656 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:25.192287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.192254 5275 generic.go:356] "Generic (PLEG): container finished" podID="2fa2010a-fed1-42f7-ad27-554fb3dc6e08" containerID="c25d9e0b3c59407602242002b940d4f90187875b64c4dd42867363c185b7c8d6" exitCode=0 Nov 03 04:02:25.192459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.192340 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" event={"ID":"2fa2010a-fed1-42f7-ad27-554fb3dc6e08","Type":"ContainerDied","Data":"c25d9e0b3c59407602242002b940d4f90187875b64c4dd42867363c185b7c8d6"} Nov 03 04:02:25.410459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.410419 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:25.413158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.413137 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:25.427972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.427942 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm"] Nov 03 04:02:25.429791 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:25.429763 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636edba7_4176_4caf_bb06_ce4a70acec15.slice/crio-4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca WatchSource:0}: Error finding container 4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca: Status 404 returned error can't find the container with id 4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca Nov 03 04:02:25.550731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.550693 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\") pod \"80924389-892f-4e3a-95ad-3a03633f8bef\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " Nov 03 04:02:25.550948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.550749 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\") pod \"80924389-892f-4e3a-95ad-3a03633f8bef\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " Nov 03 04:02:25.550948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.550792 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\" (UniqueName: \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\") pod \"5d1136b2-8585-493c-8934-64e55b3389dd\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " Nov 03 04:02:25.550948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.550920 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\" (UniqueName: \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\") pod \"80924389-892f-4e3a-95ad-3a03633f8bef\" (UID: \"80924389-892f-4e3a-95ad-3a03633f8bef\") " Nov 03 04:02:25.551145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.551082 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\") pod \"5d1136b2-8585-493c-8934-64e55b3389dd\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " Nov 03 04:02:25.551145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.551118 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\") pod \"5d1136b2-8585-493c-8934-64e55b3389dd\" (UID: \"5d1136b2-8585-493c-8934-64e55b3389dd\") " Nov 03 04:02:25.551246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.551158 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca" pod "80924389-892f-4e3a-95ad-3a03633f8bef" (UID: "80924389-892f-4e3a-95ad-3a03633f8bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:25.552160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.552122 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca" pod "5d1136b2-8585-493c-8934-64e55b3389dd" (UID: "5d1136b2-8585-493c-8934-64e55b3389dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:25.552888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.552870 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:25.553984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.553962 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:25.650142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.650100 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca" pod "80924389-892f-4e3a-95ad-3a03633f8bef" (UID: "80924389-892f-4e3a-95ad-3a03633f8bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:25.650142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.650134 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv" pod "80924389-892f-4e3a-95ad-3a03633f8bef" (UID: "80924389-892f-4e3a-95ad-3a03633f8bef"). InnerVolumeSpecName "kube-api-access-tf7sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:25.650386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.650205 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca" pod "5d1136b2-8585-493c-8934-64e55b3389dd" (UID: "5d1136b2-8585-493c-8934-64e55b3389dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:25.650386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.650310 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf" pod "5d1136b2-8585-493c-8934-64e55b3389dd" (UID: "5d1136b2-8585-493c-8934-64e55b3389dd"). InnerVolumeSpecName "kube-api-access-xssjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:25.655215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.655182 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1136b2-8585-493c-8934-64e55b3389dd-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:25.655215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.655216 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80924389-892f-4e3a-95ad-3a03633f8bef-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:25.655468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.655232 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xssjf\" (UniqueName: \"kubernetes.io/projected/5d1136b2-8585-493c-8934-64e55b3389dd-kube-api-access-xssjf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:25.655468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:25.655250 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7sv\" (UniqueName: \"kubernetes.io/projected/80924389-892f-4e3a-95ad-3a03633f8bef-kube-api-access-tf7sv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:26.202137 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.202039 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" Nov 03 04:02:26.202137 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.202042 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8" event={"ID":"5d1136b2-8585-493c-8934-64e55b3389dd","Type":"ContainerDied","Data":"cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4"} Nov 03 04:02:26.202383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.202168 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5b9a8c21cebaffc436cf6da64ccc3737f3b024f66df459902b27a585d524f4" Nov 03 04:02:26.203888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.203840 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" event={"ID":"636edba7-4176-4caf-bb06-ce4a70acec15","Type":"ContainerStarted","Data":"40f4af9d963a646722b81c57407399b13590979d855e2609c5b12471fcacb9b7"} Nov 03 04:02:26.204034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.203907 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" event={"ID":"636edba7-4176-4caf-bb06-ce4a70acec15","Type":"ContainerStarted","Data":"4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca"} Nov 03 04:02:26.205493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.205474 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" Nov 03 04:02:26.205635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.205508 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5" event={"ID":"80924389-892f-4e3a-95ad-3a03633f8bef","Type":"ContainerDied","Data":"cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1"} Nov 03 04:02:26.205635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.205551 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf656493640d7f234e86cf56cdf72aa7e3f2ee9fe1fe05e4f4bfd54e1127d8a1" Nov 03 04:02:26.322903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.322831 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" podStartSLOduration=2.322818295 podStartE2EDuration="2.322818295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:26.319727072 +0000 UTC m=+3637.472415494" watchObservedRunningTime="2026-11-03 04:02:26.322818295 +0000 UTC m=+3637.475506716" Nov 03 04:02:26.360103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.360076 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:26.465329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.465237 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\" (UniqueName: \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\") pod \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " Nov 03 04:02:26.465762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.465450 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\") pod \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " Nov 03 04:02:26.465762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.465498 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\") pod \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\" (UID: \"2fa2010a-fed1-42f7-ad27-554fb3dc6e08\") " Nov 03 04:02:26.465980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.465950 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca" pod "2fa2010a-fed1-42f7-ad27-554fb3dc6e08" (UID: "2fa2010a-fed1-42f7-ad27-554fb3dc6e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:26.467544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.467520 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9" pod "2fa2010a-fed1-42f7-ad27-554fb3dc6e08" (UID: "2fa2010a-fed1-42f7-ad27-554fb3dc6e08"). InnerVolumeSpecName "kube-api-access-trgf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:26.467694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.467676 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca" pod "2fa2010a-fed1-42f7-ad27-554fb3dc6e08" (UID: "2fa2010a-fed1-42f7-ad27-554fb3dc6e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:26.567664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.567624 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:26.567664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.567657 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:26.567664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:26.567672 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trgf9\" (UniqueName: \"kubernetes.io/projected/2fa2010a-fed1-42f7-ad27-554fb3dc6e08-kube-api-access-trgf9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:27.215049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:27.214959 5275 generic.go:356] "Generic (PLEG): container finished" podID="636edba7-4176-4caf-bb06-ce4a70acec15" containerID="40f4af9d963a646722b81c57407399b13590979d855e2609c5b12471fcacb9b7" exitCode=0 Nov 03 04:02:27.215220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:27.215044 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" event={"ID":"636edba7-4176-4caf-bb06-ce4a70acec15","Type":"ContainerDied","Data":"40f4af9d963a646722b81c57407399b13590979d855e2609c5b12471fcacb9b7"} Nov 03 04:02:27.216593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:27.216570 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" Nov 03 04:02:27.216719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:27.216598 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd" event={"ID":"2fa2010a-fed1-42f7-ad27-554fb3dc6e08","Type":"ContainerDied","Data":"96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e"} Nov 03 04:02:27.216719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:27.216640 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a01ed6372aeab5afa8ab2d6215db31ae2c4f1530f52443940b3739c94a771e" Nov 03 04:02:28.398065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.398036 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:28.493579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.493488 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\" (UniqueName: \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\") pod \"636edba7-4176-4caf-bb06-ce4a70acec15\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " Nov 03 04:02:28.493754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.493605 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\") pod \"636edba7-4176-4caf-bb06-ce4a70acec15\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " Nov 03 04:02:28.493754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.493706 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\") pod \"636edba7-4176-4caf-bb06-ce4a70acec15\" (UID: \"636edba7-4176-4caf-bb06-ce4a70acec15\") " Nov 03 04:02:28.494054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.494023 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca" pod "636edba7-4176-4caf-bb06-ce4a70acec15" (UID: "636edba7-4176-4caf-bb06-ce4a70acec15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:28.494308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.494286 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:28.496000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.495971 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca" pod "636edba7-4176-4caf-bb06-ce4a70acec15" (UID: "636edba7-4176-4caf-bb06-ce4a70acec15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:28.496175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.496158 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq" pod "636edba7-4176-4caf-bb06-ce4a70acec15" (UID: "636edba7-4176-4caf-bb06-ce4a70acec15"). InnerVolumeSpecName "kube-api-access-4rqrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:28.596098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.596063 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edba7-4176-4caf-bb06-ce4a70acec15-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:28.596098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:28.596098 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqrq\" (UniqueName: \"kubernetes.io/projected/636edba7-4176-4caf-bb06-ce4a70acec15-kube-api-access-4rqrq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:29.233443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:29.233414 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" Nov 03 04:02:29.233630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:29.233463 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm" event={"ID":"636edba7-4176-4caf-bb06-ce4a70acec15","Type":"ContainerDied","Data":"4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca"} Nov 03 04:02:29.233630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:29.233506 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4664ba641c1844b4380989936730ef51f23e102287d47a051f09628558187eca" Nov 03 04:02:32.331346 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:02:32.331304 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:02:33.171307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:33.171276 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:02:35.195618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:35.195588 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:02:39.254698 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:39.254663 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 04:02:42.055050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.055005 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:02:42.055949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.055893 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5d1136b2-8585-493c-8934-64e55b3389dd" containerName="revoke-keys" Nov 03 04:02:42.055949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.055944 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1136b2-8585-493c-8934-64e55b3389dd" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.055979 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2fa2010a-fed1-42f7-ad27-554fb3dc6e08" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.055987 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa2010a-fed1-42f7-ad27-554fb3dc6e08" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056030 5275 cpu_manager.go:414] "containerMap: removing container" podUID="636edba7-4176-4caf-bb06-ce4a70acec15" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056038 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636edba7-4176-4caf-bb06-ce4a70acec15" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056058 5275 cpu_manager.go:414] "containerMap: removing container" podUID="80924389-892f-4e3a-95ad-3a03633f8bef" containerName="revoke-keys" Nov 03 04:02:42.056156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056065 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80924389-892f-4e3a-95ad-3a03633f8bef" containerName="revoke-keys" Nov 03 04:02:42.056479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056225 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1136b2-8585-493c-8934-64e55b3389dd" containerName="revoke-keys" Nov 03 04:02:42.056479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056240 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa2010a-fed1-42f7-ad27-554fb3dc6e08" containerName="revoke-keys" Nov 03 04:02:42.056479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056254 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="636edba7-4176-4caf-bb06-ce4a70acec15" containerName="revoke-keys" Nov 03 04:02:42.056479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.056262 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="80924389-892f-4e3a-95ad-3a03633f8bef" containerName="revoke-keys" Nov 03 04:02:42.060346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.060323 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.112784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.112748 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8c832d-openshift-default-dockercfg-zhtnc\"" Nov 03 04:02:42.168274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.168242 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:02:42.193059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193025 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193061 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193233 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193262 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193287 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqnk\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193355 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193442 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193561 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.193644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.193629 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.294662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.294625 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.294662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.294661 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.294975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.294683 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqnk\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.294975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.294904 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.294975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.294951 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295026 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295083 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295089 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295114 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295139 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295184 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295399 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295551 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.295723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.295704 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.297335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.297311 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.297623 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.297604 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.391996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.388500 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.391996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.389593 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqnk\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\") pod \"e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:42.676466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:42.676375 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:43.142969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.142919 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:02:43.148944 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:43.148908 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc671a7_f4ac_47f6_81e6_79194593da5a.slice/crio-efb7ed7ed78e2a62500f161fcff19ab2ff0fd9166e01c1b4a2071621f1c14a89 WatchSource:0}: Error finding container efb7ed7ed78e2a62500f161fcff19ab2ff0fd9166e01c1b4a2071621f1c14a89: Status 404 returned error can't find the container with id efb7ed7ed78e2a62500f161fcff19ab2ff0fd9166e01c1b4a2071621f1c14a89 Nov 03 04:02:43.151562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.151523 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:43.151653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.151596 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:43.151653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.151630 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:43.410068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.409955 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerStarted","Data":"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6"} Nov 03 04:02:43.410068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.410030 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerStarted","Data":"efb7ed7ed78e2a62500f161fcff19ab2ff0fd9166e01c1b4a2071621f1c14a89"} Nov 03 04:02:43.530743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.530681 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podStartSLOduration=1.5306627499999998 podStartE2EDuration="1.53066275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:43.529540057 +0000 UTC m=+3654.682228478" watchObservedRunningTime="2026-11-03 04:02:43.53066275 +0000 UTC m=+3654.683351172" Nov 03 04:02:43.676566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:43.676487 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:44.683032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:44.682955 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:45.427097 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:45.427061 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:45.427981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:45.427956 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:02:46.079359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:46.079320 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 04:02:48.242799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.242745 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:02:48.253070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.250476 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.303790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.303757 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0afad-openshift-default-dockercfg-gq52h\"" Nov 03 04:02:48.359064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.359031 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:02:48.387695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.387658 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.387938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.387782 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.387938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.387826 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bbk\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.387938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.387889 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.388076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.388032 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.388076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.388059 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.388161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.388087 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.388227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.388209 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.388261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.388248 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.489719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489685 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.489719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489732 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bbk\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489761 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489873 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489904 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.489945 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490014 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490054 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490091 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490453 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490691 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490743 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490791 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.490966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.490943 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.492450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.492430 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.492901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.492806 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.551136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.551105 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.551321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.551256 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bbk\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\") pod \"e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:48.573088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:48.573038 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:49.008711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.008685 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:02:49.008964 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:49.008941 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f020784_16fc_48bc_aca4_bd86707cecb1.slice/crio-93515b5006916bad6a6b6ceb59ee6ad6db960ce61bc0c9c440cd9003471471c0 WatchSource:0}: Error finding container 93515b5006916bad6a6b6ceb59ee6ad6db960ce61bc0c9c440cd9003471471c0: Status 404 returned error can't find the container with id 93515b5006916bad6a6b6ceb59ee6ad6db960ce61bc0c9c440cd9003471471c0 Nov 03 04:02:49.011474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.011440 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:49.011561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.011507 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:49.011561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.011536 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:02:49.465728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.465693 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerStarted","Data":"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33"} Nov 03 04:02:49.466203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.465789 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerStarted","Data":"93515b5006916bad6a6b6ceb59ee6ad6db960ce61bc0c9c440cd9003471471c0"} Nov 03 04:02:49.574304 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.574270 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:49.590311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:49.590259 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podStartSLOduration=1.59024513 podStartE2EDuration="1.59024513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:49.586689474 +0000 UTC m=+3660.739377900" watchObservedRunningTime="2026-11-03 04:02:49.59024513 +0000 UTC m=+3660.742933550" Nov 03 04:02:50.581342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:50.581312 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:51.483540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:51.483509 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:51.484513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:51.484490 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:02:53.147130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.147103 5275 scope.go:122] "RemoveContainer" containerID="6f783bb344398ec9d59754d755cc1ad8c47c292ce99426d1787607e7c02ce3cc" Nov 03 04:02:53.884514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.884478 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:02:53.890277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.890257 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:53.946701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.946669 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:02:53.996871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.992430 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c832d-dockercfg-xz77b\"" Nov 03 04:02:53.996871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:53.992566 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c832d\"" Nov 03 04:02:54.009869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.007262 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.009869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.007469 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62dz\" (UniqueName: \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.108786 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.108749 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62dz\" (UniqueName: \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.109052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.109020 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.109841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.109819 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.172392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.172324 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62dz\" (UniqueName: \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\") pod \"payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.204379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.204346 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:54.281145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.279971 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" podUID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" containerName="main" containerID="cri-o://82d2049da4db0c3d533889406299a4fa67ff89a3e543e8fc6664e65967b1e27a" gracePeriod=45 Nov 03 04:02:54.513820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.513788 5275 generic.go:356] "Generic (PLEG): container finished" podID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" containerID="82d2049da4db0c3d533889406299a4fa67ff89a3e543e8fc6664e65967b1e27a" exitCode=0 Nov 03 04:02:54.514021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.513996 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" event={"ID":"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8","Type":"ContainerDied","Data":"82d2049da4db0c3d533889406299a4fa67ff89a3e543e8fc6664e65967b1e27a"} Nov 03 04:02:54.597406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.597377 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:02:54.598172 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:54.598148 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c80bd4_17f0_49eb_b3f6_73a652ee4407.slice/crio-313dac6d1b8f922bd3dd0f24f9aefbf02c78ba6c5c176e1213c5c4de108d3f2f WatchSource:0}: Error finding container 313dac6d1b8f922bd3dd0f24f9aefbf02c78ba6c5c176e1213c5c4de108d3f2f: Status 404 returned error can't find the container with id 313dac6d1b8f922bd3dd0f24f9aefbf02c78ba6c5c176e1213c5c4de108d3f2f Nov 03 04:02:54.600255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.600231 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 04:02:54.717480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717445 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.717756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717517 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.717756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717560 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.717756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717610 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.717756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717640 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.717756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717681 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\" (UniqueName: \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\") pod \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\" (UID: \"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8\") " Nov 03 04:02:54.718119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717815 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:54.718119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.717824 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:54.718300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.718173 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:54.718822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.718790 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:54.718822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.718813 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:54.719014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.718829 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:54.720575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.720549 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:54.720678 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.720618 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:54.721546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.721525 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7" pod "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" (UID: "6953518f-c01f-45d1-bb44-b7cc9bfa9ef8"). InnerVolumeSpecName "kube-api-access-pl7f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:54.820468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.820431 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:54.820468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.820461 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:54.820468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:54.820470 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7f7\" (UniqueName: \"kubernetes.io/projected/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8-kube-api-access-pl7f7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:55.522507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.522481 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" Nov 03 04:02:55.522936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.522522 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5" event={"ID":"6953518f-c01f-45d1-bb44-b7cc9bfa9ef8","Type":"ContainerDied","Data":"599a8b21a07fd26357977f6942fd22e0b53a41ac7edf44ef2ea7a680c8ee9d45"} Nov 03 04:02:55.522936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.522572 5275 scope.go:122] "RemoveContainer" containerID="82d2049da4db0c3d533889406299a4fa67ff89a3e543e8fc6664e65967b1e27a" Nov 03 04:02:55.524559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.524522 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" event={"ID":"64c80bd4-17f0-49eb-b3f6-73a652ee4407","Type":"ContainerStarted","Data":"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10"} Nov 03 04:02:55.524668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.524591 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:02:55.524668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.524610 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" event={"ID":"64c80bd4-17f0-49eb-b3f6-73a652ee4407","Type":"ContainerStarted","Data":"313dac6d1b8f922bd3dd0f24f9aefbf02c78ba6c5c176e1213c5c4de108d3f2f"} Nov 03 04:02:55.689727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.689691 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 04:02:55.739452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.739420 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9daccc/auth-test-model-9daccc-kserve-7697dc9dd9-pg6m5"] Nov 03 04:02:55.855857 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:55.855732 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" podStartSLOduration=2.855711749 podStartE2EDuration="2.855711749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:55.85196835 +0000 UTC m=+3667.004656776" watchObservedRunningTime="2026-11-03 04:02:55.855711749 +0000 UTC m=+3667.008400172" Nov 03 04:02:56.985492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:56.985459 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" path="/var/lib/kubelet/pods/6953518f-c01f-45d1-bb44-b7cc9bfa9ef8/volumes" Nov 03 04:02:58.725256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.725185 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:02:58.725999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.725981 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" containerName="main" Nov 03 04:02:58.726067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.726000 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" containerName="main" Nov 03 04:02:58.726138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.726129 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6953518f-c01f-45d1-bb44-b7cc9bfa9ef8" containerName="main" Nov 03 04:02:58.731221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.731202 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.774904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.774867 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh79n\" (UniqueName: \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.775096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.775078 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.789295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.789122 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:02:58.804873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.802186 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0afad\"" Nov 03 04:02:58.841885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.835840 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0afad-dockercfg-5h9ld\"" Nov 03 04:02:58.877898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.867663 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:02:58.877898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.873276 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:58.896635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.894534 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.896635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.894696 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh79n\" (UniqueName: \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.896635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.896303 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.927511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.927473 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:02:58.958513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.958472 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh79n\" (UniqueName: \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\") pod \"payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:58.996382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.996298 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:58.996659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:58.996633 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.044599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.044564 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:59.098739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.098524 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.098739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.098638 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.099666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.099640 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.159932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.159786 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\") pod \"payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.202445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.202418 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:02:59.443425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.443399 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:02:59.444023 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:59.444002 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda131f172_9465_4944_8035_9b5834047220.slice/crio-4823a4aa16d7816e3b922d450a645ccc9ca9b74f7c141dc35139394da41bb3a0 WatchSource:0}: Error finding container 4823a4aa16d7816e3b922d450a645ccc9ca9b74f7c141dc35139394da41bb3a0: Status 404 returned error can't find the container with id 4823a4aa16d7816e3b922d450a645ccc9ca9b74f7c141dc35139394da41bb3a0 Nov 03 04:02:59.558500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.558457 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" event={"ID":"a131f172-9465-4944-8035-9b5834047220","Type":"ContainerStarted","Data":"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44"} Nov 03 04:02:59.558670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.558527 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" event={"ID":"a131f172-9465-4944-8035-9b5834047220","Type":"ContainerStarted","Data":"4823a4aa16d7816e3b922d450a645ccc9ca9b74f7c141dc35139394da41bb3a0"} Nov 03 04:02:59.558670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.558556 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:02:59.592930 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:02:59.592900 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634dba1a_4097_4e1a_b2f6_79870477f52b.slice/crio-e05f2d643bd1991b20056ee274a17565cbbff49d4f566b89c2816e3e22d80e03 WatchSource:0}: Error finding container e05f2d643bd1991b20056ee274a17565cbbff49d4f566b89c2816e3e22d80e03: Status 404 returned error can't find the container with id e05f2d643bd1991b20056ee274a17565cbbff49d4f566b89c2816e3e22d80e03 Nov 03 04:02:59.593724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.593691 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:02:59.679505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:02:59.679440 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" podStartSLOduration=1.679419572 podStartE2EDuration="1.679419572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:59.674039148 +0000 UTC m=+3670.826727569" watchObservedRunningTime="2026-11-03 04:02:59.679419572 +0000 UTC m=+3670.832107994" Nov 03 04:03:00.572409 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:00.572366 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" event={"ID":"634dba1a-4097-4e1a-b2f6-79870477f52b","Type":"ContainerStarted","Data":"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5"} Nov 03 04:03:00.572917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:00.572457 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:03:00.572917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:00.572479 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" event={"ID":"634dba1a-4097-4e1a-b2f6-79870477f52b","Type":"ContainerStarted","Data":"e05f2d643bd1991b20056ee274a17565cbbff49d4f566b89c2816e3e22d80e03"} Nov 03 04:03:00.692156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:00.692094 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" podStartSLOduration=2.692079621 podStartE2EDuration="2.692079621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:00.687950578 +0000 UTC m=+3671.840638999" watchObservedRunningTime="2026-11-03 04:03:00.692079621 +0000 UTC m=+3671.844768083" Nov 03 04:03:01.107962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.107920 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" podUID="aec54fe2-3251-4ab7-ac22-4021dc21f032" containerName="main" containerID="cri-o://b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e" gracePeriod=45 Nov 03 04:03:01.400916 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.400882 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 04:03:01.530142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530110 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530200 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\" (UniqueName: \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530231 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530257 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530317 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:01.530489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530433 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\" (UniqueName: \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530483 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\") pod \"aec54fe2-3251-4ab7-ac22-4021dc21f032\" (UID: \"aec54fe2-3251-4ab7-ac22-4021dc21f032\") " Nov 03 04:03:01.530767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530598 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:01.530835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.530766 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:01.531166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.531143 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.531253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.531173 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.531253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.531212 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.532754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.532726 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:01.533005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.532987 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "kube-api-access-d5thk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:01.533182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.533149 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs" pod "aec54fe2-3251-4ab7-ac22-4021dc21f032" (UID: "aec54fe2-3251-4ab7-ac22-4021dc21f032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:03:01.582741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.582705 5275 generic.go:356] "Generic (PLEG): container finished" podID="aec54fe2-3251-4ab7-ac22-4021dc21f032" containerID="b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e" exitCode=0 Nov 03 04:03:01.583261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.582761 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" Nov 03 04:03:01.583261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.582791 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" event={"ID":"aec54fe2-3251-4ab7-ac22-4021dc21f032","Type":"ContainerDied","Data":"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e"} Nov 03 04:03:01.583261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.582882 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch" event={"ID":"aec54fe2-3251-4ab7-ac22-4021dc21f032","Type":"ContainerDied","Data":"e2ca414c588e1752f1b78e3db4c304458806bf0fa279a455ca31dac758839e02"} Nov 03 04:03:01.583261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.582915 5275 scope.go:122] "RemoveContainer" containerID="b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e" Nov 03 04:03:01.596907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.596887 5275 scope.go:122] "RemoveContainer" containerID="b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e" Nov 03 04:03:01.597247 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:03:01.597226 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e\": container with ID starting with b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e not found: ID does not exist" containerID="b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e" Nov 03 04:03:01.597308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.597257 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e"} err="failed to get container status \"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e\": rpc error: code = NotFound desc = could not find container \"b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e\": container with ID starting with b321010e9b9d2f17ad4e71487b19a164d7d7222c0607e6417425927acdd6c67e not found: ID does not exist" Nov 03 04:03:01.632257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.632165 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec54fe2-3251-4ab7-ac22-4021dc21f032-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.632257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.632199 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5thk\" (UniqueName: \"kubernetes.io/projected/aec54fe2-3251-4ab7-ac22-4021dc21f032-kube-api-access-d5thk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.632257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.632210 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec54fe2-3251-4ab7-ac22-4021dc21f032-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:01.748304 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.748264 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 04:03:01.754753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:01.754715 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7ed2d6/auth-test-model-7ed2d6-kserve-6cc97fdb89-zccch"] Nov 03 04:03:02.331273 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:03:02.331240 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:03:02.986662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:02.986627 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec54fe2-3251-4ab7-ac22-4021dc21f032" path="/var/lib/kubelet/pods/aec54fe2-3251-4ab7-ac22-4021dc21f032/volumes" Nov 03 04:03:06.536553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:06.536520 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:03:10.576323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.576285 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:03:10.931145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.931053 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:03:10.932021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.932000 5275 cpu_manager.go:414] "containerMap: removing container" podUID="aec54fe2-3251-4ab7-ac22-4021dc21f032" containerName="main" Nov 03 04:03:10.932094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.932024 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec54fe2-3251-4ab7-ac22-4021dc21f032" containerName="main" Nov 03 04:03:10.932240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.932227 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec54fe2-3251-4ab7-ac22-4021dc21f032" containerName="main" Nov 03 04:03:10.937885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.937823 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:10.991279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.991228 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-66d902-openshift-default-dockercfg-5zpwp\"" Nov 03 04:03:10.999331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:10.999298 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:03:11.093188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093150 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093205 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093368 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093454 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093633 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093676 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093746 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzdx\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093800 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.093956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.093935 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.194893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.194779 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.194893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.194835 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzdx\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.194912 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.194946 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.194982 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195031 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195126 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195220 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195265 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195452 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195552 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195575 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195886 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.195998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.195960 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.198264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.198057 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.198264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.198226 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.256761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.256729 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzdx\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.256980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.256780 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.256980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.256965 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:11.586713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.586681 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:03:11.734514 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:03:11.734487 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa17b61_8bab_4fa0_a556_6dd919a79ec2.slice/crio-b2f5b067fa92e92f213f5077e9ed9dadd64375abe7eecf6e8cd4a3d9407ed034 WatchSource:0}: Error finding container b2f5b067fa92e92f213f5077e9ed9dadd64375abe7eecf6e8cd4a3d9407ed034: Status 404 returned error can't find the container with id b2f5b067fa92e92f213f5077e9ed9dadd64375abe7eecf6e8cd4a3d9407ed034 Nov 03 04:03:11.735647 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.735621 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:03:11.738044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.738003 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:11.738165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.738088 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:11.738165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:11.738129 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:12.688661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:12.688625 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" event={"ID":"4aa17b61-8bab-4fa0-a556-6dd919a79ec2","Type":"ContainerStarted","Data":"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796"} Nov 03 04:03:12.689122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:12.688677 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" event={"ID":"4aa17b61-8bab-4fa0-a556-6dd919a79ec2","Type":"ContainerStarted","Data":"b2f5b067fa92e92f213f5077e9ed9dadd64375abe7eecf6e8cd4a3d9407ed034"} Nov 03 04:03:12.809500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:12.809399 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podStartSLOduration=2.809379134 podStartE2EDuration="2.809379134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:12.808127253 +0000 UTC m=+3683.960815675" watchObservedRunningTime="2026-11-03 04:03:12.809379134 +0000 UTC m=+3683.962067558" Nov 03 04:03:13.260237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:13.260201 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:13.264993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:13.264949 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 03 04:03:13.265274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:13.265247 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 03 04:03:14.267883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:14.267343 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 03 04:03:14.267883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:14.267424 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 03 04:03:15.267873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:15.263006 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 03 04:03:15.267873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:15.263070 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 03 04:03:16.275077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:16.275038 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 03 04:03:16.275680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:16.275646 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 03 04:03:17.268872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:17.268253 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 03 04:03:17.268872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:17.268318 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 03 04:03:18.262680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:18.262647 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:18.263180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:18.263155 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:18.263987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:18.263966 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:03:23.751913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.751819 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:03:23.757321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.757296 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790478 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790720 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790555 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790720 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790600 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790722 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790775 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.790883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790837 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.791077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790908 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9z6p\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.791077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790950 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.791077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.790977 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.809395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.809366 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-90139c-openshift-default-dockercfg-f2m4z\"" Nov 03 04:03:23.865111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.865074 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:03:23.911944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.902765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.912355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.912325 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9z6p\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.913061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.910492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.913285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.913264 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.913899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.913877 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.914342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.914320 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.915284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.915251 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.915419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.915343 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.915520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.915500 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.915596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.915546 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.915700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.913798 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.916178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.916155 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.916343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.916282 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.917214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.917186 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.921228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.921201 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.921816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.921797 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.973610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.973577 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9z6p\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:23.974834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:23.974802 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\") pod \"e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:24.075419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.075388 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:24.516841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.516794 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:03:24.517174 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:03:24.517148 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad62fb63_8fe6_49d5_b3a9_86da11f61230.slice/crio-96f8b74f105c4c84adcfa750db3e7e44586859dcc2ddf0d55f464a29ee6c5710 WatchSource:0}: Error finding container 96f8b74f105c4c84adcfa750db3e7e44586859dcc2ddf0d55f464a29ee6c5710: Status 404 returned error can't find the container with id 96f8b74f105c4c84adcfa750db3e7e44586859dcc2ddf0d55f464a29ee6c5710 Nov 03 04:03:24.519988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.519950 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:24.520097 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.520030 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:24.520097 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.520077 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:24.783068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.783004 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerStarted","Data":"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f"} Nov 03 04:03:24.783595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.783092 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerStarted","Data":"96f8b74f105c4c84adcfa750db3e7e44586859dcc2ddf0d55f464a29ee6c5710"} Nov 03 04:03:24.910318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:24.910254 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podStartSLOduration=1.91023481 podStartE2EDuration="1.91023481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:24.90464343 +0000 UTC m=+3696.057331853" watchObservedRunningTime="2026-11-03 04:03:24.91023481 +0000 UTC m=+3696.062923231" Nov 03 04:03:25.076651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:25.076571 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:25.082944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:25.082916 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:25.794989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:25.794960 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:25.796222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:25.796198 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:03:32.331632 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:03:32.331595 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:03:37.874476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:37.874434 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 04:03:39.474827 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.474787 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:03:39.480579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.480557 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.542242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.542211 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-55dc0c-openshift-default-dockercfg-jdmrt\"" Nov 03 04:03:39.587216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.587183 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:03:39.629040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.628997 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt9w\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.629418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.629394 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.629642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.629624 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.629772 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.629757 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.630332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.630310 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.630613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.630582 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.630714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.630643 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.630714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.630706 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.630841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.630788 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732412 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732462 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732481 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732516 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732542 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732581 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732652 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732682 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt9w\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.732811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.732741 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.733155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.733061 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.733155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.733136 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.733269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.733214 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.733430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.733358 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.733509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.733447 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.735069 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.735039 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.735227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.735212 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.802306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.802280 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:39.802473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:39.802282 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt9w\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\") pod \"e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:40.095627 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.095589 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:40.545741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.545707 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:03:40.546642 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:03:40.546611 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbda912_9cf0_416c_99a2_5536aaee1db5.slice/crio-b547871cf3f7f961fb56301769c5e9a8e1ea4a2ebd817ff3b01822c303faf138 WatchSource:0}: Error finding container b547871cf3f7f961fb56301769c5e9a8e1ea4a2ebd817ff3b01822c303faf138: Status 404 returned error can't find the container with id b547871cf3f7f961fb56301769c5e9a8e1ea4a2ebd817ff3b01822c303faf138 Nov 03 04:03:40.549687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.549648 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:40.549799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.549721 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:40.549799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.549753 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:03:40.928615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.928518 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerStarted","Data":"69c6f4e24eb30efa8a8231d5561c8a7b16a34218319d958f9bcff9402ccea24e"} Nov 03 04:03:40.928615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:40.928577 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerStarted","Data":"b547871cf3f7f961fb56301769c5e9a8e1ea4a2ebd817ff3b01822c303faf138"} Nov 03 04:03:41.052554 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:41.052501 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podStartSLOduration=2.052487728 podStartE2EDuration="2.052487728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:41.049781714 +0000 UTC m=+3712.202470164" watchObservedRunningTime="2026-11-03 04:03:41.052487728 +0000 UTC m=+3712.205176148" Nov 03 04:03:41.096672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:41.096627 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:41.102654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:41.102627 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:41.937839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:41.937788 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:41.938971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:41.938948 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:03:48.665480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:48.665439 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 04:03:49.031426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.031383 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:03:49.070485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.070455 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.087123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.087091 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:03:49.123949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.123914 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-55dc0c-dockercfg-fwsbb\"" Nov 03 04:03:49.123949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.123942 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-55dc0c\"" Nov 03 04:03:49.161688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.161634 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nb4\" (UniqueName: \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.161945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.161917 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.263797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.263754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nb4\" (UniqueName: \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.263980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.263936 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.315318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.315246 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-55dc0c\"" Nov 03 04:03:49.325475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.325439 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nb4\" (UniqueName: \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.325653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.325601 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\") pod \"payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.402265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.402222 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:49.803815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:49.803774 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:03:49.804195 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:03:49.803943 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d5bd27_9451_4b02_9491_f462d6d9ff10.slice/crio-ad5897f96c88e9f913e39ead0444f2bb7af13660806ed9b5d06b9e180fd48ec7 WatchSource:0}: Error finding container ad5897f96c88e9f913e39ead0444f2bb7af13660806ed9b5d06b9e180fd48ec7: Status 404 returned error can't find the container with id ad5897f96c88e9f913e39ead0444f2bb7af13660806ed9b5d06b9e180fd48ec7 Nov 03 04:03:50.061485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:50.061382 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" event={"ID":"63d5bd27-9451-4b02-9491-f462d6d9ff10","Type":"ContainerStarted","Data":"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e"} Nov 03 04:03:50.061485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:50.061451 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" event={"ID":"63d5bd27-9451-4b02-9491-f462d6d9ff10","Type":"ContainerStarted","Data":"ad5897f96c88e9f913e39ead0444f2bb7af13660806ed9b5d06b9e180fd48ec7"} Nov 03 04:03:50.061485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:50.061486 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:03:50.180629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:50.180565 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" podStartSLOduration=2.180545963 podStartE2EDuration="2.180545963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:50.177896452 +0000 UTC m=+3721.330584874" watchObservedRunningTime="2026-11-03 04:03:50.180545963 +0000 UTC m=+3721.333234384" Nov 03 04:03:52.896095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:52.896034 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerName="main" containerID="cri-o://b95bc3b5274046fee7a06eee12e0b8737ccd19849151a976a49dabc440d645be" gracePeriod=45 Nov 03 04:03:53.089294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.089252 5275 generic.go:356] "Generic (PLEG): container finished" podID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerID="b95bc3b5274046fee7a06eee12e0b8737ccd19849151a976a49dabc440d645be" exitCode=0 Nov 03 04:03:53.089497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.089291 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" event={"ID":"13b2efa7-9f97-44c3-83d9-888c4828d2d7","Type":"ContainerDied","Data":"b95bc3b5274046fee7a06eee12e0b8737ccd19849151a976a49dabc440d645be"} Nov 03 04:03:53.190211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.190190 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 04:03:53.230696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.230666 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.230905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.230724 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.230905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.230785 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\" (UniqueName: \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.230905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.230830 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.231093 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.230995 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.231093 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231057 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:53.231213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231159 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\") pod \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\" (UID: \"13b2efa7-9f97-44c3-83d9-888c4828d2d7\") " Nov 03 04:03:53.231213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231183 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:53.231403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231383 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:53.231709 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231692 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:53.231782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231709 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:53.231782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.231718 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:53.233173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.233142 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:53.233254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.233196 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "kube-api-access-ttspz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:53.233254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.233201 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs" pod "13b2efa7-9f97-44c3-83d9-888c4828d2d7" (UID: "13b2efa7-9f97-44c3-83d9-888c4828d2d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:03:53.333098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.333068 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttspz\" (UniqueName: \"kubernetes.io/projected/13b2efa7-9f97-44c3-83d9-888c4828d2d7-kube-api-access-ttspz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:53.333098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.333093 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b2efa7-9f97-44c3-83d9-888c4828d2d7-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:53.333098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:53.333104 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b2efa7-9f97-44c3-83d9-888c4828d2d7-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:54.098458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.098417 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" Nov 03 04:03:54.098458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.098436 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" event={"ID":"13b2efa7-9f97-44c3-83d9-888c4828d2d7","Type":"ContainerDied","Data":"b45116c4e7b79073a42a73fc53eef20577e4c7bff6d2adb5149ca87ecdc49292"} Nov 03 04:03:54.099144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.098489 5275 scope.go:122] "RemoveContainer" containerID="b95bc3b5274046fee7a06eee12e0b8737ccd19849151a976a49dabc440d645be" Nov 03 04:03:54.152773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.152730 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.232:8000/ready\": context deadline exceeded" Nov 03 04:03:54.268823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.268786 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 04:03:54.272773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.272746 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/auth-test-model-3d8dc5-kserve-676b8bd77-4cxvb"] Nov 03 04:03:54.985997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:03:54.985965 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" path="/var/lib/kubelet/pods/13b2efa7-9f97-44c3-83d9-888c4828d2d7/volumes" Nov 03 04:04:01.071589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:01.071561 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:04:02.334686 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:04:02.334636 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:04:03.687585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:03.687519 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" podUID="e2faa216-ec0c-4e1b-a201-20f489fe3406" containerName="main" containerID="cri-o://167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36" gracePeriod=45 Nov 03 04:04:04.082472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.082437 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 04:04:04.198742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.198707 5275 generic.go:356] "Generic (PLEG): container finished" podID="e2faa216-ec0c-4e1b-a201-20f489fe3406" containerID="167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36" exitCode=0 Nov 03 04:04:04.198992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.198746 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" event={"ID":"e2faa216-ec0c-4e1b-a201-20f489fe3406","Type":"ContainerDied","Data":"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36"} Nov 03 04:04:04.198992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.198882 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" event={"ID":"e2faa216-ec0c-4e1b-a201-20f489fe3406","Type":"ContainerDied","Data":"b656ff32a6da6712952a8105caa5520e495202cdde392f43f1ce6eeb9ab1c9b8"} Nov 03 04:04:04.198992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.198763 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f" Nov 03 04:04:04.198992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.198911 5275 scope.go:122] "RemoveContainer" containerID="167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36" Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200070 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200309 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200359 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200591 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200668 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200729 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\" (UniqueName: \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\") pod \"e2faa216-ec0c-4e1b-a201-20f489fe3406\" (UID: \"e2faa216-ec0c-4e1b-a201-20f489fe3406\") " Nov 03 04:04:04.200929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.200818 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:04:04.201528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.201413 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.201972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.201871 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:04:04.202090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.202017 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:04:04.204711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.204648 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "kube-api-access-tqqrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:04:04.204711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.204656 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:04:04.205039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.205019 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm" pod "e2faa216-ec0c-4e1b-a201-20f489fe3406" (UID: "e2faa216-ec0c-4e1b-a201-20f489fe3406"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:04:04.243932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.243906 5275 scope.go:122] "RemoveContainer" containerID="167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36" Nov 03 04:04:04.244263 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:04:04.244242 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36\": container with ID starting with 167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36 not found: ID does not exist" containerID="167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36" Nov 03 04:04:04.244363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.244273 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36"} err="failed to get container status \"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36\": rpc error: code = NotFound desc = could not find container \"167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36\": container with ID starting with 167b426e65666344c35872cd598e54accefd0db17f3d7b16d6eda027db599d36 not found: ID does not exist" Nov 03 04:04:04.302767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.302722 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.302767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.302757 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.303125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.302788 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faa216-ec0c-4e1b-a201-20f489fe3406-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.303125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.302798 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faa216-ec0c-4e1b-a201-20f489fe3406-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.303125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.302809 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqrk\" (UniqueName: \"kubernetes.io/projected/e2faa216-ec0c-4e1b-a201-20f489fe3406-kube-api-access-tqqrk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:04:04.621698 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.621660 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 04:04:04.625220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.625197 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-286a8c/auth-test-model-286a8c-kserve-59b596645-5xj5f"] Nov 03 04:04:04.985989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:04.985896 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2faa216-ec0c-4e1b-a201-20f489fe3406" path="/var/lib/kubelet/pods/e2faa216-ec0c-4e1b-a201-20f489fe3406/volumes" Nov 03 04:04:05.208562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.208524 5275 generic.go:356] "Generic (PLEG): container finished" podID="cc5db47a-df8b-4df0-8d97-112092993e04" containerID="2b32c04d53f6f345aa3b537021856a40cd06320e7be7c3cabbd7e60a949eb192" exitCode=0 Nov 03 04:04:05.208763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.208599 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerDied","Data":"2b32c04d53f6f345aa3b537021856a40cd06320e7be7c3cabbd7e60a949eb192"} Nov 03 04:04:05.209152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.209126 5275 scope.go:122] "RemoveContainer" containerID="2b32c04d53f6f345aa3b537021856a40cd06320e7be7c3cabbd7e60a949eb192" Nov 03 04:04:05.210952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.210925 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:05.211013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.210995 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:05.211054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:05.211023 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:06.220207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:06.220167 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerStarted","Data":"b8a222380361996d2831931b7f9f3d522e35d34831c0c0606ac85ce93e44ebd8"} Nov 03 04:04:06.579222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:06.579179 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:04:06.579555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:06.579528 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 04:04:06.579662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:06.579587 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 04:04:07.578829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:07.578791 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 04:04:07.579275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:07.578875 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 04:04:08.579546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:08.579511 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 04:04:08.580118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:08.579578 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 04:04:09.579483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:09.579441 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 04:04:09.579694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:09.579511 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 04:04:10.578817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:10.578786 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 04:04:10.579036 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:10.578840 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 04:04:11.583513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:11.583483 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:04:11.584064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:11.584038 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:04:11.584860 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:11.584821 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:04:32.331362 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:04:32.331327 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:04:37.532472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.532440 5275 generic.go:356] "Generic (PLEG): container finished" podID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerID="62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6" exitCode=0 Nov 03 04:04:37.532982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.532515 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerDied","Data":"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6"} Nov 03 04:04:37.532982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.532967 5275 scope.go:122] "RemoveContainer" containerID="62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6" Nov 03 04:04:37.534679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.534653 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:37.534744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.534723 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:37.534797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:37.534756 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:04:38.542421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:38.542383 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerStarted","Data":"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd"} Nov 03 04:04:38.679076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:38.679042 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:04:38.679717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:38.679692 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:38.679980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:38.679953 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:39.685876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:39.681223 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:39.685876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:39.681287 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:40.679770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:40.679286 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:40.679770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:40.679350 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:41.677420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:41.677382 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:41.677838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:41.677437 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:42.677913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:42.677869 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:42.678353 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:42.677944 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:43.677794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:43.677760 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Nov 03 04:04:43.678002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:43.677834 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Nov 03 04:04:44.681929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:44.681828 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:04:44.682385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:44.682367 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:04:44.683115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:04:44.683095 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:05:02.330473 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:05:02.330436 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:05:12.864540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.864453 5275 generic.go:356] "Generic (PLEG): container finished" podID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerID="09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33" exitCode=0 Nov 03 04:05:12.864540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.864528 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerDied","Data":"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33"} Nov 03 04:05:12.865122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.865019 5275 scope.go:122] "RemoveContainer" containerID="09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33" Nov 03 04:05:12.866748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.866720 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:12.866809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.866793 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:12.866870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:12.866829 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:13.888134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:13.888095 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerStarted","Data":"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044"} Nov 03 04:05:14.576340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:14.576306 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:05:14.576737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:14.576710 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 03 04:05:14.576954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:14.576922 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 03 04:05:15.579998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:15.579954 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 03 04:05:15.580518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:15.580018 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 03 04:05:16.574301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:16.574261 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 03 04:05:16.574572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:16.574539 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 03 04:05:17.575319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:17.575277 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 03 04:05:17.576016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:17.575981 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 03 04:05:18.574335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:18.574294 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 03 04:05:18.574628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:18.574599 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 03 04:05:19.580438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:19.580400 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:05:19.581030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:19.580761 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:05:19.581664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:19.581636 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:05:25.752126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:25.752088 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 04:05:25.862301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:25.862263 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 04:05:30.753013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:30.752967 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" podUID="b5b788c3-9f2f-4255-a302-c1e80b37449b" containerName="payload-processing" containerID="cri-o://dde64ffe23359696ca9aad0ad435497ee774e9ca761d83c41e4ddb976fd81b0a" gracePeriod=25 Nov 03 04:05:30.862815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:30.862771 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" podUID="88e6eebe-0057-4893-adc6-204f5a9d0143" containerName="payload-pre-processing" containerID="cri-o://ff7d35ba4f6cd6240a7b9456ae5361a74458e6fe9fc068c7384e73ce2e3796c8" gracePeriod=25 Nov 03 04:05:31.026766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.026732 5275 generic.go:356] "Generic (PLEG): container finished" podID="88e6eebe-0057-4893-adc6-204f5a9d0143" containerID="ff7d35ba4f6cd6240a7b9456ae5361a74458e6fe9fc068c7384e73ce2e3796c8" exitCode=0 Nov 03 04:05:31.026964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.026807 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" event={"ID":"88e6eebe-0057-4893-adc6-204f5a9d0143","Type":"ContainerDied","Data":"ff7d35ba4f6cd6240a7b9456ae5361a74458e6fe9fc068c7384e73ce2e3796c8"} Nov 03 04:05:31.028567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.028542 5275 generic.go:356] "Generic (PLEG): container finished" podID="b5b788c3-9f2f-4255-a302-c1e80b37449b" containerID="dde64ffe23359696ca9aad0ad435497ee774e9ca761d83c41e4ddb976fd81b0a" exitCode=0 Nov 03 04:05:31.028710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.028612 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" event={"ID":"b5b788c3-9f2f-4255-a302-c1e80b37449b","Type":"ContainerDied","Data":"dde64ffe23359696ca9aad0ad435497ee774e9ca761d83c41e4ddb976fd81b0a"} Nov 03 04:05:31.028710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.028658 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" event={"ID":"b5b788c3-9f2f-4255-a302-c1e80b37449b","Type":"ContainerDied","Data":"e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3"} Nov 03 04:05:31.028710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.028679 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d654d6221a52002f88f12ab08328589348b405d40bb8a2a185ef96238494f3" Nov 03 04:05:31.033923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.033900 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 04:05:31.040380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.040358 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\") pod \"b5b788c3-9f2f-4255-a302-c1e80b37449b\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " Nov 03 04:05:31.040552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.040530 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\" (UniqueName: \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\") pod \"b5b788c3-9f2f-4255-a302-c1e80b37449b\" (UID: \"b5b788c3-9f2f-4255-a302-c1e80b37449b\") " Nov 03 04:05:31.040724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.040702 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume" pod "b5b788c3-9f2f-4255-a302-c1e80b37449b" (UID: "b5b788c3-9f2f-4255-a302-c1e80b37449b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:05:31.041141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.041124 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b788c3-9f2f-4255-a302-c1e80b37449b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:31.042751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.042725 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m" pod "b5b788c3-9f2f-4255-a302-c1e80b37449b" (UID: "b5b788c3-9f2f-4255-a302-c1e80b37449b"). InnerVolumeSpecName "kube-api-access-72t6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:05:31.132606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.132580 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 04:05:31.142455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.142430 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\" (UniqueName: \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\") pod \"88e6eebe-0057-4893-adc6-204f5a9d0143\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " Nov 03 04:05:31.142602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.142482 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\") pod \"88e6eebe-0057-4893-adc6-204f5a9d0143\" (UID: \"88e6eebe-0057-4893-adc6-204f5a9d0143\") " Nov 03 04:05:31.142904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.142873 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume" pod "88e6eebe-0057-4893-adc6-204f5a9d0143" (UID: "88e6eebe-0057-4893-adc6-204f5a9d0143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:05:31.143032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.142923 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72t6m\" (UniqueName: \"kubernetes.io/projected/b5b788c3-9f2f-4255-a302-c1e80b37449b-kube-api-access-72t6m\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:31.144777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.144759 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl" pod "88e6eebe-0057-4893-adc6-204f5a9d0143" (UID: "88e6eebe-0057-4893-adc6-204f5a9d0143"). InnerVolumeSpecName "kube-api-access-ss9gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:05:31.243924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.243881 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9gl\" (UniqueName: \"kubernetes.io/projected/88e6eebe-0057-4893-adc6-204f5a9d0143-kube-api-access-ss9gl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:31.243924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:31.243929 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e6eebe-0057-4893-adc6-204f5a9d0143-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:32.037681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.037651 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" Nov 03 04:05:32.038274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.037711 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j" event={"ID":"88e6eebe-0057-4893-adc6-204f5a9d0143","Type":"ContainerDied","Data":"a7008c1c779aad25ffbd3d0b1d9dce064b6ae31a34133097451fd20e9753b678"} Nov 03 04:05:32.038274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.037800 5275 scope.go:122] "RemoveContainer" containerID="ff7d35ba4f6cd6240a7b9456ae5361a74458e6fe9fc068c7384e73ce2e3796c8" Nov 03 04:05:32.038274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.037935 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs" Nov 03 04:05:32.204057 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.204015 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 04:05:32.254521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.254487 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5172b1-644d44fc55-v5t5j"] Nov 03 04:05:32.323986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.323901 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 04:05:32.327413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.327383 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5172b1-f8b449c79-hpngs"] Nov 03 04:05:32.336954 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:05:32.336917 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:05:32.985346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.985311 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e6eebe-0057-4893-adc6-204f5a9d0143" path="/var/lib/kubelet/pods/88e6eebe-0057-4893-adc6-204f5a9d0143/volumes" Nov 03 04:05:32.985746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:32.985730 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b788c3-9f2f-4255-a302-c1e80b37449b" path="/var/lib/kubelet/pods/b5b788c3-9f2f-4255-a302-c1e80b37449b/volumes" Nov 03 04:05:52.197418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:52.197378 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:05:53.231746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:53.231705 5275 scope.go:122] "RemoveContainer" containerID="dde64ffe23359696ca9aad0ad435497ee774e9ca761d83c41e4ddb976fd81b0a" Nov 03 04:05:56.250487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.250442 5275 generic.go:356] "Generic (PLEG): container finished" podID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerID="69c6f4e24eb30efa8a8231d5561c8a7b16a34218319d958f9bcff9402ccea24e" exitCode=0 Nov 03 04:05:56.251076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.250473 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerDied","Data":"69c6f4e24eb30efa8a8231d5561c8a7b16a34218319d958f9bcff9402ccea24e"} Nov 03 04:05:56.251076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.251065 5275 scope.go:122] "RemoveContainer" containerID="69c6f4e24eb30efa8a8231d5561c8a7b16a34218319d958f9bcff9402ccea24e" Nov 03 04:05:56.252939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.252906 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:56.253038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.252982 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:56.253038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.253021 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:05:56.938719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:56.938682 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:05:57.198247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.198158 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" podUID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" containerName="payload-pre-processing" containerID="cri-o://44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10" gracePeriod=25 Nov 03 04:05:57.285335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.285297 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:05:57.291184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.291152 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerStarted","Data":"8417279e265aa0b7fa1576e7d6e33402c08f2f58c0ec75949bde83e7af755ca3"} Nov 03 04:05:57.405253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.404996 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:05:57.610275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.610246 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:05:57.813649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.806500 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\" (UniqueName: \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\") pod \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " Nov 03 04:05:57.813649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.806588 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\") pod \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\" (UID: \"64c80bd4-17f0-49eb-b3f6-73a652ee4407\") " Nov 03 04:05:57.813649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.810142 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume" pod "64c80bd4-17f0-49eb-b3f6-73a652ee4407" (UID: "64c80bd4-17f0-49eb-b3f6-73a652ee4407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:05:57.827245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.826374 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz" pod "64c80bd4-17f0-49eb-b3f6-73a652ee4407" (UID: "64c80bd4-17f0-49eb-b3f6-73a652ee4407"). InnerVolumeSpecName "kube-api-access-h62dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:05:57.908206 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.908173 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62dz\" (UniqueName: \"kubernetes.io/projected/64c80bd4-17f0-49eb-b3f6-73a652ee4407-kube-api-access-h62dz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:57.908206 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:57.908200 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c80bd4-17f0-49eb-b3f6-73a652ee4407-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:58.096743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.096645 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:05:58.098072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.098040 5275 patch_prober.go:29] interesting pod/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 03 04:05:58.098214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.098102 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 03 04:05:58.310921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.308757 5275 generic.go:356] "Generic (PLEG): container finished" podID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" containerID="44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10" exitCode=0 Nov 03 04:05:58.310921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.308814 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" Nov 03 04:05:58.310921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.308899 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" event={"ID":"64c80bd4-17f0-49eb-b3f6-73a652ee4407","Type":"ContainerDied","Data":"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10"} Nov 03 04:05:58.310921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.308945 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn" event={"ID":"64c80bd4-17f0-49eb-b3f6-73a652ee4407","Type":"ContainerDied","Data":"313dac6d1b8f922bd3dd0f24f9aefbf02c78ba6c5c176e1213c5c4de108d3f2f"} Nov 03 04:05:58.310921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.308972 5275 scope.go:122] "RemoveContainer" containerID="44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10" Nov 03 04:05:58.329545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.329499 5275 scope.go:122] "RemoveContainer" containerID="44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10" Nov 03 04:05:58.329972 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:05:58.329946 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10\": container with ID starting with 44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10 not found: ID does not exist" containerID="44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10" Nov 03 04:05:58.330080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.329979 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10"} err="failed to get container status \"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10\": rpc error: code = NotFound desc = could not find container \"44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10\": container with ID starting with 44296b659dc0baaa19aebb130d2d39112a708460e666a07f43e0af3ed8ff8e10 not found: ID does not exist" Nov 03 04:05:58.510006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.509904 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:05:58.543873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.543713 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c832d-55c78ddb4c-pg2qn"] Nov 03 04:05:58.990716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:58.990682 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" path="/var/lib/kubelet/pods/64c80bd4-17f0-49eb-b3f6-73a652ee4407/volumes" Nov 03 04:05:59.096581 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:59.096544 5275 patch_prober.go:29] interesting pod/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 03 04:05:59.096770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:05:59.096623 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 03 04:06:00.096226 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:00.096184 5275 patch_prober.go:29] interesting pod/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 03 04:06:00.096617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:00.096268 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 03 04:06:01.096030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:01.095996 5275 patch_prober.go:29] interesting pod/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 03 04:06:01.096237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:01.096057 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 03 04:06:02.096615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.096571 5275 patch_prober.go:29] interesting pod/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 03 04:06:02.097220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.096630 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 03 04:06:02.286276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.286205 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" podUID="634dba1a-4097-4e1a-b2f6-79870477f52b" containerName="payload-processing" containerID="cri-o://64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5" gracePeriod=25 Nov 03 04:06:02.332440 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:02.332394 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:06:02.405692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.405653 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" podUID="a131f172-9465-4944-8035-9b5834047220" containerName="payload-pre-processing" containerID="cri-o://d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44" gracePeriod=25 Nov 03 04:06:02.605881 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.605100 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:06:02.727131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.727104 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\") pod \"634dba1a-4097-4e1a-b2f6-79870477f52b\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " Nov 03 04:06:02.727841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.727237 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\") pod \"634dba1a-4097-4e1a-b2f6-79870477f52b\" (UID: \"634dba1a-4097-4e1a-b2f6-79870477f52b\") " Nov 03 04:06:02.728035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.728003 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume" pod "634dba1a-4097-4e1a-b2f6-79870477f52b" (UID: "634dba1a-4097-4e1a-b2f6-79870477f52b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:02.730517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.730337 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7" pod "634dba1a-4097-4e1a-b2f6-79870477f52b" (UID: "634dba1a-4097-4e1a-b2f6-79870477f52b"). InnerVolumeSpecName "kube-api-access-4ptt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:02.743100 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.743074 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:06:02.828062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828029 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\" (UniqueName: \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\") pod \"a131f172-9465-4944-8035-9b5834047220\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " Nov 03 04:06:02.828244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828187 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\") pod \"a131f172-9465-4944-8035-9b5834047220\" (UID: \"a131f172-9465-4944-8035-9b5834047220\") " Nov 03 04:06:02.828555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828527 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume" pod "a131f172-9465-4944-8035-9b5834047220" (UID: "a131f172-9465-4944-8035-9b5834047220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:02.828713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828698 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634dba1a-4097-4e1a-b2f6-79870477f52b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:02.828765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828716 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/634dba1a-4097-4e1a-b2f6-79870477f52b-kube-api-access-4ptt7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:02.828765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.828725 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a131f172-9465-4944-8035-9b5834047220-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:02.830279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.830255 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n" pod "a131f172-9465-4944-8035-9b5834047220" (UID: "a131f172-9465-4944-8035-9b5834047220"). InnerVolumeSpecName "kube-api-access-xh79n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:02.930536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:02.930498 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh79n\" (UniqueName: \"kubernetes.io/projected/a131f172-9465-4944-8035-9b5834047220-kube-api-access-xh79n\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:03.101389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.101357 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:06:03.101885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.101809 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:06:03.102644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.102625 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:06:03.358838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.352742 5275 generic.go:356] "Generic (PLEG): container finished" podID="634dba1a-4097-4e1a-b2f6-79870477f52b" containerID="64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5" exitCode=0 Nov 03 04:06:03.358838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.352993 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" event={"ID":"634dba1a-4097-4e1a-b2f6-79870477f52b","Type":"ContainerDied","Data":"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5"} Nov 03 04:06:03.358838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.353055 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" event={"ID":"634dba1a-4097-4e1a-b2f6-79870477f52b","Type":"ContainerDied","Data":"e05f2d643bd1991b20056ee274a17565cbbff49d4f566b89c2816e3e22d80e03"} Nov 03 04:06:03.358838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.353085 5275 scope.go:122] "RemoveContainer" containerID="64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5" Nov 03 04:06:03.358838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.353260 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2" Nov 03 04:06:03.359750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.359156 5275 generic.go:356] "Generic (PLEG): container finished" podID="a131f172-9465-4944-8035-9b5834047220" containerID="d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44" exitCode=0 Nov 03 04:06:03.361392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.360061 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" Nov 03 04:06:03.361392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.360201 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" event={"ID":"a131f172-9465-4944-8035-9b5834047220","Type":"ContainerDied","Data":"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44"} Nov 03 04:06:03.361392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.360252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82" event={"ID":"a131f172-9465-4944-8035-9b5834047220","Type":"ContainerDied","Data":"4823a4aa16d7816e3b922d450a645ccc9ca9b74f7c141dc35139394da41bb3a0"} Nov 03 04:06:03.372973 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.372623 5275 scope.go:122] "RemoveContainer" containerID="64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5" Nov 03 04:06:03.373396 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:03.373243 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5\": container with ID starting with 64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5 not found: ID does not exist" containerID="64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5" Nov 03 04:06:03.373396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.373281 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5"} err="failed to get container status \"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5\": rpc error: code = NotFound desc = could not find container \"64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5\": container with ID starting with 64453b703b8c488bfdd930ac0a0f92045c627a752a376c70d6b935f63abe38f5 not found: ID does not exist" Nov 03 04:06:03.373396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.373307 5275 scope.go:122] "RemoveContainer" containerID="d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44" Nov 03 04:06:03.392416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.392038 5275 scope.go:122] "RemoveContainer" containerID="d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44" Nov 03 04:06:03.392775 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:03.392683 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44\": container with ID starting with d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44 not found: ID does not exist" containerID="d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44" Nov 03 04:06:03.392775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.392722 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44"} err="failed to get container status \"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44\": rpc error: code = NotFound desc = could not find container \"d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44\": container with ID starting with d161e94679b4ffa98ccc5ce00d1e1c70309bb4571cfc8d3f18f88886a85c5d44 not found: ID does not exist" Nov 03 04:06:03.520647 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.520611 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:06:03.524137 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.524111 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0afad-685d7f9ff9-rct82"] Nov 03 04:06:03.631265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.631183 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:06:03.638439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:03.638408 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0afad-6b87f9dd57-8xsm2"] Nov 03 04:06:04.985805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:04.985771 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634dba1a-4097-4e1a-b2f6-79870477f52b" path="/var/lib/kubelet/pods/634dba1a-4097-4e1a-b2f6-79870477f52b/volumes" Nov 03 04:06:04.986240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:04.986225 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a131f172-9465-4944-8035-9b5834047220" path="/var/lib/kubelet/pods/a131f172-9465-4944-8035-9b5834047220/volumes" Nov 03 04:06:06.669086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:06.669053 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:06:06.669522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:06.669272 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" containerID="cri-o://9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044" gracePeriod=30 Nov 03 04:06:11.948024 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:11.947955 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:06:12.050908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.050872 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.050920 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.050980 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051068 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051096 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051130 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051159 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051243 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051287 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\") pod \"2f020784-16fc-48bc-aca4-bd86707cecb1\" (UID: \"2f020784-16fc-48bc-aca4-bd86707cecb1\") " Nov 03 04:06:12.051519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051236 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:12.051519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051337 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:12.051519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051440 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:12.051954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051901 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:12.052055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.052033 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.052125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.052063 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.052125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.052080 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.052125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.052094 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.052369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.051625 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:12.053512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.053488 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:06:12.053512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.053497 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "kube-api-access-84bbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:12.054263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.054240 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:12.054387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.054319 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token" pod "2f020784-16fc-48bc-aca4-bd86707cecb1" (UID: "2f020784-16fc-48bc-aca4-bd86707cecb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:12.153430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.153391 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.153430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.153421 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.153430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.153432 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f020784-16fc-48bc-aca4-bd86707cecb1-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.153430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.153441 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.153823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.153452 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bbk\" (UniqueName: \"kubernetes.io/projected/2f020784-16fc-48bc-aca4-bd86707cecb1-kube-api-access-84bbk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:12.435385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.435347 5275 generic.go:356] "Generic (PLEG): container finished" podID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerID="9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044" exitCode=0 Nov 03 04:06:12.435615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.435426 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" Nov 03 04:06:12.435615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.435430 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerDied","Data":"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044"} Nov 03 04:06:12.435615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.435521 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p" event={"ID":"2f020784-16fc-48bc-aca4-bd86707cecb1","Type":"ContainerDied","Data":"93515b5006916bad6a6b6ceb59ee6ad6db960ce61bc0c9c440cd9003471471c0"} Nov 03 04:06:12.435615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.435537 5275 scope.go:122] "RemoveContainer" containerID="9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044" Nov 03 04:06:12.447271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.447251 5275 scope.go:122] "RemoveContainer" containerID="09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33" Nov 03 04:06:12.464496 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.464473 5275 scope.go:122] "RemoveContainer" containerID="9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044" Nov 03 04:06:12.464893 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:12.464862 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044\": container with ID starting with 9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044 not found: ID does not exist" containerID="9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044" Nov 03 04:06:12.465022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.464902 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044"} err="failed to get container status \"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044\": rpc error: code = NotFound desc = could not find container \"9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044\": container with ID starting with 9310805c9123c9f1135bad68004b8bfabf6a1f4dfeb34689a282f0b6c4ca6044 not found: ID does not exist" Nov 03 04:06:12.465022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.464926 5275 scope.go:122] "RemoveContainer" containerID="09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33" Nov 03 04:06:12.465253 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:12.465232 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33\": container with ID starting with 09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33 not found: ID does not exist" containerID="09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33" Nov 03 04:06:12.465301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.465262 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33"} err="failed to get container status \"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33\": rpc error: code = NotFound desc = could not find container \"09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33\": container with ID starting with 09759316c93afd61b61e1af1000a38b7880679597d59c34a62d9b6a950e61e33 not found: ID does not exist" Nov 03 04:06:12.602151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.602118 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:06:12.605408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.605382 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0afad-openshift-default-7866776d89-tqf8p"] Nov 03 04:06:12.985255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:12.985224 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" path="/var/lib/kubelet/pods/2f020784-16fc-48bc-aca4-bd86707cecb1/volumes" Nov 03 04:06:15.814384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.814346 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:06:15.815179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815156 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815196 5275 cpu_manager.go:414] "containerMap: removing container" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerName="main" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815204 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerName="main" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815217 5275 cpu_manager.go:414] "containerMap: removing container" podUID="634dba1a-4097-4e1a-b2f6-79870477f52b" containerName="payload-processing" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815225 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634dba1a-4097-4e1a-b2f6-79870477f52b" containerName="payload-processing" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815254 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b5b788c3-9f2f-4255-a302-c1e80b37449b" containerName="payload-processing" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815262 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b788c3-9f2f-4255-a302-c1e80b37449b" containerName="payload-processing" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815280 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a131f172-9465-4944-8035-9b5834047220" containerName="payload-pre-processing" Nov 03 04:06:15.815299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815288 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a131f172-9465-4944-8035-9b5834047220" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815322 5275 cpu_manager.go:414] "containerMap: removing container" podUID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815330 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815344 5275 cpu_manager.go:414] "containerMap: removing container" podUID="88e6eebe-0057-4893-adc6-204f5a9d0143" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815351 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e6eebe-0057-4893-adc6-204f5a9d0143" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815368 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815375 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815392 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e2faa216-ec0c-4e1b-a201-20f489fe3406" containerName="main" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815400 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2faa216-ec0c-4e1b-a201-20f489fe3406" containerName="main" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815572 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b788c3-9f2f-4255-a302-c1e80b37449b" containerName="payload-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815586 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a131f172-9465-4944-8035-9b5834047220" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815601 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c80bd4-17f0-49eb-b3f6-73a652ee4407" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815613 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e6eebe-0057-4893-adc6-204f5a9d0143" containerName="payload-pre-processing" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815624 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815633 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2faa216-ec0c-4e1b-a201-20f489fe3406" containerName="main" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815646 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b2efa7-9f97-44c3-83d9-888c4828d2d7" containerName="main" Nov 03 04:06:15.815782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.815654 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="634dba1a-4097-4e1a-b2f6-79870477f52b" containerName="payload-processing" Nov 03 04:06:15.819209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.819188 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:15.873477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.873389 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 04:06:15.922794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.922724 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 04:06:15.922794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.922749 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 04:06:15.922794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.922737 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-de22ce75b964\"" Nov 03 04:06:15.922794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.922784 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-sqwqb\"" Nov 03 04:06:15.923275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.923240 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 04:06:15.926449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:15.926426 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:06:16.005866 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.005821 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.006122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.006099 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bkq\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.006204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.006152 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.006286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.006259 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.108194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.108108 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bkq\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.108194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.108157 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.108441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.108246 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.108441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.108325 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.109075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.109048 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.109252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.109166 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.111380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.111353 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.169676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.169644 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bkq\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\") pod \"payload-processing-external-model-7c9b799cc-6jgjw\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.433392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.433299 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:16.816132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:16.816098 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:06:16.816468 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:06:16.816400 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b797ad9_d283_461c_ad0c_692d992c722c.slice/crio-e0b2cd7a93daff526c9d0d507cac52a8ed630571c3120713b9e957872d9f3347 WatchSource:0}: Error finding container e0b2cd7a93daff526c9d0d507cac52a8ed630571c3120713b9e957872d9f3347: Status 404 returned error can't find the container with id e0b2cd7a93daff526c9d0d507cac52a8ed630571c3120713b9e957872d9f3347 Nov 03 04:06:17.479648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:17.479601 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" event={"ID":"2b797ad9-d283-461c-ad0c-692d992c722c","Type":"ContainerStarted","Data":"4154827367ea0e456d28b7a599f0f713de7c2fb0127e61d2482da38709328936"} Nov 03 04:06:17.479899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:17.479674 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:17.479899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:17.479700 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" event={"ID":"2b797ad9-d283-461c-ad0c-692d992c722c","Type":"ContainerStarted","Data":"e0b2cd7a93daff526c9d0d507cac52a8ed630571c3120713b9e957872d9f3347"} Nov 03 04:06:17.597892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:17.597807 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" podStartSLOduration=2.597789793 podStartE2EDuration="2.597789793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:06:17.595619993 +0000 UTC m=+3868.748308415" watchObservedRunningTime="2026-11-03 04:06:17.597789793 +0000 UTC m=+3868.750478229" Nov 03 04:06:28.494087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:28.493979 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:06:31.605470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:31.605436 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 04:06:31.605894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:31.605697 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" containerID="cri-o://566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87" gracePeriod=30 Nov 03 04:06:32.333059 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:32.333020 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:06:36.918295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:36.918264 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:06:37.016316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016277 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016349 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016403 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016520 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016558 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016590 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016611 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:37.016891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016644 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016674 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.016891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016770 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\") pod \"e89da8fb-d241-443a-a119-0be1bfef9c2b\" (UID: \"e89da8fb-d241-443a-a119-0be1bfef9c2b\") " Nov 03 04:06:37.017061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016892 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:37.017061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.016896 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:37.017159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017108 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:37.017217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017186 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:37.017621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017599 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.017697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017628 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.017697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017644 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.017697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017660 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.017697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.017674 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89da8fb-d241-443a-a119-0be1bfef9c2b-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.018928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.018908 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:06:37.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.019135 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "kube-api-access-p8b8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:37.019391 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.019373 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:06:37.019453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.019430 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token" pod "e89da8fb-d241-443a-a119-0be1bfef9c2b" (UID: "e89da8fb-d241-443a-a119-0be1bfef9c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:37.118755 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.118702 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.118755 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.118748 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.119054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.118766 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8b8n\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-kube-api-access-p8b8n\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.119054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.118781 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89da8fb-d241-443a-a119-0be1bfef9c2b-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:37.693217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.693186 5275 generic.go:356] "Generic (PLEG): container finished" podID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerID="566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87" exitCode=0 Nov 03 04:06:37.693412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.693254 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" Nov 03 04:06:37.693412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.693274 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerDied","Data":"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87"} Nov 03 04:06:37.693412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.693340 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf" event={"ID":"e89da8fb-d241-443a-a119-0be1bfef9c2b","Type":"ContainerDied","Data":"578bcc5c09152a3392a84b35a45ebbd20e8af0912dfdc1892914f4feba8df38d"} Nov 03 04:06:37.693412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.693372 5275 scope.go:122] "RemoveContainer" containerID="566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87" Nov 03 04:06:37.705446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.705421 5275 scope.go:122] "RemoveContainer" containerID="6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c" Nov 03 04:06:37.715995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.715971 5275 scope.go:122] "RemoveContainer" containerID="566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87" Nov 03 04:06:37.716388 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:37.716363 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87\": container with ID starting with 566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87 not found: ID does not exist" containerID="566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87" Nov 03 04:06:37.716447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.716399 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87"} err="failed to get container status \"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87\": rpc error: code = NotFound desc = could not find container \"566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87\": container with ID starting with 566bd24bc37835f2bf1c3a59e20d00f71562d29723b2824296aaae8151fb6e87 not found: ID does not exist" Nov 03 04:06:37.716447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.716420 5275 scope.go:122] "RemoveContainer" containerID="6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c" Nov 03 04:06:37.716722 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:37.716703 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c\": container with ID starting with 6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c not found: ID does not exist" containerID="6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c" Nov 03 04:06:37.716789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.716728 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c"} err="failed to get container status \"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c\": rpc error: code = NotFound desc = could not find container \"6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c\": container with ID starting with 6a7aaab98ad004b10a7ac885948fc665ff26acd2d46e17076d3ff6da7bd36d9c not found: ID does not exist" Nov 03 04:06:37.842419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.842386 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:06:37.908988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.908953 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 04:06:37.912296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.912270 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5172b1-openshift-default-894cdbd6f-ft6nf"] Nov 03 04:06:37.987889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:37.987764 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:06:38.990880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:38.990830 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" path="/var/lib/kubelet/pods/e89da8fb-d241-443a-a119-0be1bfef9c2b/volumes" Nov 03 04:06:41.837066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:41.837024 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:06:41.837525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:41.837098 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:06:42.843019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:42.842955 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" podUID="2bd963bc-1b69-42ee-b90e-100e8155defb" containerName="payload-processing" containerID="cri-o://0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739" gracePeriod=25 Nov 03 04:06:42.988495 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:42.988447 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerName="payload-pre-processing" containerID="cri-o://3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5" gracePeriod=25 Nov 03 04:06:43.155011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.154972 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:06:43.168040 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:06:43.168009 5275 logging.go:55] [core] [Channel #10432 SubChannel #10433]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.255:50052", ServerName: "10.133.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.255:50052: connect: connection refused" Nov 03 04:06:43.225236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.225195 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\") pod \"2bd963bc-1b69-42ee-b90e-100e8155defb\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " Nov 03 04:06:43.225476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.225446 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\" (UniqueName: \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\") pod \"2bd963bc-1b69-42ee-b90e-100e8155defb\" (UID: \"2bd963bc-1b69-42ee-b90e-100e8155defb\") " Nov 03 04:06:43.226617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.226587 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume" pod "2bd963bc-1b69-42ee-b90e-100e8155defb" (UID: "2bd963bc-1b69-42ee-b90e-100e8155defb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:43.229000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.228968 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5" pod "2bd963bc-1b69-42ee-b90e-100e8155defb" (UID: "2bd963bc-1b69-42ee-b90e-100e8155defb"). InnerVolumeSpecName "kube-api-access-2v4w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:43.284491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.284442 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:06:43.327511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.327478 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4w5\" (UniqueName: \"kubernetes.io/projected/2bd963bc-1b69-42ee-b90e-100e8155defb-kube-api-access-2v4w5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:43.327511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.327508 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd963bc-1b69-42ee-b90e-100e8155defb-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:43.428889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.428766 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\") pod \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " Nov 03 04:06:43.429096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.428892 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\" (UniqueName: \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\") pod \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\" (UID: \"4af9a49c-e2ac-4936-905d-fdb6d88232b4\") " Nov 03 04:06:43.429200 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.429174 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume" pod "4af9a49c-e2ac-4936-905d-fdb6d88232b4" (UID: "4af9a49c-e2ac-4936-905d-fdb6d88232b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:43.429432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.429414 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af9a49c-e2ac-4936-905d-fdb6d88232b4-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:43.431917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.431884 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp" pod "4af9a49c-e2ac-4936-905d-fdb6d88232b4" (UID: "4af9a49c-e2ac-4936-905d-fdb6d88232b4"). InnerVolumeSpecName "kube-api-access-gw6gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:43.530535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.530499 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6gp\" (UniqueName: \"kubernetes.io/projected/4af9a49c-e2ac-4936-905d-fdb6d88232b4-kube-api-access-gw6gp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:43.749924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.749798 5275 generic.go:356] "Generic (PLEG): container finished" podID="2bd963bc-1b69-42ee-b90e-100e8155defb" containerID="0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739" exitCode=0 Nov 03 04:06:43.749924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.749892 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" Nov 03 04:06:43.750333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.749887 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" event={"ID":"2bd963bc-1b69-42ee-b90e-100e8155defb","Type":"ContainerDied","Data":"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739"} Nov 03 04:06:43.750333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.750029 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh" event={"ID":"2bd963bc-1b69-42ee-b90e-100e8155defb","Type":"ContainerDied","Data":"f82591dc51275ffb1adc317943caf5a1df6900922df4d62c26295bbd664b222d"} Nov 03 04:06:43.750333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.750061 5275 scope.go:122] "RemoveContainer" containerID="0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739" Nov 03 04:06:43.752267 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.752178 5275 generic.go:356] "Generic (PLEG): container finished" podID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerID="3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5" exitCode=0 Nov 03 04:06:43.752267 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.752235 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" Nov 03 04:06:43.752484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.752235 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" event={"ID":"4af9a49c-e2ac-4936-905d-fdb6d88232b4","Type":"ContainerDied","Data":"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5"} Nov 03 04:06:43.752484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.752362 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" event={"ID":"4af9a49c-e2ac-4936-905d-fdb6d88232b4","Type":"ContainerDied","Data":"6111c41639d07e12289a0867ed969361e4e2a6541cd2c09a04fa59f2d51ce295"} Nov 03 04:06:43.766570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.766531 5275 scope.go:122] "RemoveContainer" containerID="0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739" Nov 03 04:06:43.767359 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:43.767330 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739\": container with ID starting with 0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739 not found: ID does not exist" containerID="0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739" Nov 03 04:06:43.767492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.767369 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739"} err="failed to get container status \"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739\": rpc error: code = NotFound desc = could not find container \"0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739\": container with ID starting with 0773eaf7e6d8370c844e07e0e5a52e2f9bbb0cc1c4c0b94e2088a81f39c61739 not found: ID does not exist" Nov 03 04:06:43.767492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.767395 5275 scope.go:122] "RemoveContainer" containerID="3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5" Nov 03 04:06:43.782009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.781978 5275 scope.go:122] "RemoveContainer" containerID="3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5" Nov 03 04:06:43.782392 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:43.782361 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5\": container with ID starting with 3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5 not found: ID does not exist" containerID="3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5" Nov 03 04:06:43.782497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.782401 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5"} err="failed to get container status \"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5\": rpc error: code = NotFound desc = could not find container \"3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5\": container with ID starting with 3c2cc9b069dda35ec31f579e0a3513638c63f1323afd8687677734fcf3dd51c5 not found: ID does not exist" Nov 03 04:06:43.915403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.915374 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:06:44.037336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:43.966626 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03da9e-7569c465cc-nntxh"] Nov 03 04:06:44.037336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.027375 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:06:44.037336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.031238 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57"] Nov 03 04:06:44.168307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.168259 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03da9e-5b7988f5ff-jhg57" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.255:50052\" within 1s: context deadline exceeded" Nov 03 04:06:44.168511 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:06:44.168366 5275 logging.go:55] [core] [Channel #10432 SubChannel #10433]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.255:50052", ServerName: "10.133.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.255:50052: operation was canceled" Nov 03 04:06:44.764004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.763911 5275 generic.go:356] "Generic (PLEG): container finished" podID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerID="79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f" exitCode=0 Nov 03 04:06:44.764004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.763981 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerDied","Data":"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f"} Nov 03 04:06:44.764514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.764480 5275 scope.go:122] "RemoveContainer" containerID="79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f" Nov 03 04:06:44.766144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.766114 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:44.766260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.766200 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:44.766260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.766248 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:44.989648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.989612 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd963bc-1b69-42ee-b90e-100e8155defb" path="/var/lib/kubelet/pods/2bd963bc-1b69-42ee-b90e-100e8155defb/volumes" Nov 03 04:06:44.990343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:44.990248 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" path="/var/lib/kubelet/pods/4af9a49c-e2ac-4936-905d-fdb6d88232b4/volumes" Nov 03 04:06:45.783822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:45.783782 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerStarted","Data":"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675"} Nov 03 04:06:46.076908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:46.076305 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:06:46.076908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:46.076431 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:46.076908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:46.076473 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:47.076482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.076444 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:47.076672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.076504 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:47.642666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.642628 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:06:47.643311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643293 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerName="payload-pre-processing" Nov 03 04:06:47.643358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643315 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerName="payload-pre-processing" Nov 03 04:06:47.643358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643344 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.643358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643350 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.643460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643360 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2bd963bc-1b69-42ee-b90e-100e8155defb" containerName="payload-processing" Nov 03 04:06:47.643460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643367 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd963bc-1b69-42ee-b90e-100e8155defb" containerName="payload-processing" Nov 03 04:06:47.643525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643469 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f020784-16fc-48bc-aca4-bd86707cecb1" containerName="istio-proxy" Nov 03 04:06:47.643525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643477 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.643525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643485 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af9a49c-e2ac-4936-905d-fdb6d88232b4" containerName="payload-pre-processing" Nov 03 04:06:47.643525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643494 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.643525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.643501 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd963bc-1b69-42ee-b90e-100e8155defb" containerName="payload-processing" Nov 03 04:06:47.645906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.645890 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.684228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.682434 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf"] Nov 03 04:06:47.684228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.683184 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.684228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.683204 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89da8fb-d241-443a-a119-0be1bfef9c2b" containerName="istio-proxy" Nov 03 04:06:47.686171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.686149 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.697999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.697972 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\"" Nov 03 04:06:47.761575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.761534 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:06:47.809050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809063 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809091 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809140 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809222 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809307 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809360 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809431 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809474 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809507 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809531 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809556 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4fq\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-kube-api-access-wh4fq\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809605 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809635 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.809705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809660 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.810085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809703 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.810085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809744 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cfb\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.810085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809768 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.810085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.809988 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf"] Nov 03 04:06:47.910556 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910440 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910556 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910518 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910579 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910664 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910705 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910728 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910755 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4fq\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-kube-api-access-wh4fq\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.910828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910832 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910890 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910930 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910971 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cfb\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.910998 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911027 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911036 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911086 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911095 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911138 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911156 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.911233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911216 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.912105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911311 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.912105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911411 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.912105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911427 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.912105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911793 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.912105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.911882 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.912408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.912126 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.912408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.912264 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.912408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.912331 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.913656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.913633 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.913809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.913631 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.913936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.913877 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.914258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.914242 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.975954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.975921 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cfb\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.976535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.976504 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4fq\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-kube-api-access-wh4fq\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.976711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.976643 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xq4ld\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:47.976711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.976712 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c9c1c8-26c9-4b51-abb9-a86c517ac991-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nxgbf\" (UID: \"21c9c1c8-26c9-4b51-abb9-a86c517ac991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:47.983249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:47.983221 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:48.001088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.000979 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:48.083315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.083277 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:48.083528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.083335 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:48.474282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.474249 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:06:48.479118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.478916 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf"] Nov 03 04:06:48.485826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.485700 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.485968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.485867 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.485968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.485944 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.488512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.488059 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.488512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.488155 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.488512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.488214 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:06:48.817956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.817907 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" event={"ID":"21c9c1c8-26c9-4b51-abb9-a86c517ac991","Type":"ContainerStarted","Data":"2fd3811eb9c9a2237e121c3b6355f33a6dd8b240a678c79775c3946e8ace7efe"} Nov 03 04:06:48.818488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.818027 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" event={"ID":"21c9c1c8-26c9-4b51-abb9-a86c517ac991","Type":"ContainerStarted","Data":"09d896771913f1f906765956e2720efd4e6bf551c6b0e7d08ece7d99936a3a8c"} Nov 03 04:06:48.819956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.819925 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" event={"ID":"adb2ca5e-d035-4d57-840c-13686d47cdb8","Type":"ContainerStarted","Data":"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5"} Nov 03 04:06:48.820115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.819975 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" event={"ID":"adb2ca5e-d035-4d57-840c-13686d47cdb8","Type":"ContainerStarted","Data":"c522745c44053c6c78afc61c470026794aac0a8d88506181e4a0e91c97d22d58"} Nov 03 04:06:48.942577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.942512 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podStartSLOduration=1.942493888 podStartE2EDuration="1.942493888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:06:48.942204244 +0000 UTC m=+3900.094892678" watchObservedRunningTime="2026-11-03 04:06:48.942493888 +0000 UTC m=+3900.095182310" Nov 03 04:06:48.992909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:48.992876 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:49.002004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.001969 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:49.003721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.003693 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:49.003920 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.003740 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:49.083937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.083656 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:49.083937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.083719 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:49.084688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.084639 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podStartSLOduration=2.084621902 podStartE2EDuration="2.084621902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:06:49.084113288 +0000 UTC m=+3900.236801710" watchObservedRunningTime="2026-11-03 04:06:49.084621902 +0000 UTC m=+3900.237310321" Nov 03 04:06:49.987873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.986142 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:49.987873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:49.986197 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:50.011871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.006293 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:50.011871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.006360 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:50.076873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.076234 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:50.076873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.076299 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:50.244789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.244711 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:06:50.244789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.244745 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:06:50.646686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.646639 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:06:50.987934 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.987214 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:50.987934 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:50.987277 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:51.046833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.042598 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:51.046833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.042672 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:51.084871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.082024 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 04:06:51.084871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.082084 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 04:06:51.993873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.992957 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:51.993873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:51.993022 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:52.009970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.008967 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:52.009970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.009047 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:52.137222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.137185 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:06:52.139164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.139138 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:06:52.141079 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.141053 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:06:52.990873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.987141 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:52.991212 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:52.991172 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:53.002041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:53.002003 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:53.002571 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:53.002082 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:53.999317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:53.999275 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:53.999661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:53.999622 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:54.006468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:54.006308 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:54.006468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:54.006382 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:54.983863 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:54.983807 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:54.984049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:54.983886 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:55.002241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.002202 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:55.002424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.002261 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:55.647893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.647827 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" podUID="63d5bd27-9451-4b02-9491-f462d6d9ff10" containerName="payload-processing" containerID="cri-o://6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e" gracePeriod=25 Nov 03 04:06:55.948064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.946234 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:06:55.952900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.952840 5275 generic.go:356] "Generic (PLEG): container finished" podID="63d5bd27-9451-4b02-9491-f462d6d9ff10" containerID="6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e" exitCode=0 Nov 03 04:06:55.953062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.952953 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" Nov 03 04:06:55.953062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.953007 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" event={"ID":"63d5bd27-9451-4b02-9491-f462d6d9ff10","Type":"ContainerDied","Data":"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e"} Nov 03 04:06:55.953220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.953096 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd" event={"ID":"63d5bd27-9451-4b02-9491-f462d6d9ff10","Type":"ContainerDied","Data":"ad5897f96c88e9f913e39ead0444f2bb7af13660806ed9b5d06b9e180fd48ec7"} Nov 03 04:06:55.953220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.953120 5275 scope.go:122] "RemoveContainer" containerID="6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e" Nov 03 04:06:55.963886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.963859 5275 scope.go:122] "RemoveContainer" containerID="6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e" Nov 03 04:06:55.964253 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:06:55.964220 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e\": container with ID starting with 6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e not found: ID does not exist" containerID="6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e" Nov 03 04:06:55.964363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.964264 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e"} err="failed to get container status \"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e\": rpc error: code = NotFound desc = could not find container \"6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e\": container with ID starting with 6babca5b764523dd64559ebb9a757375e94616edbf4da9247cd6d5668879b88e not found: ID does not exist" Nov 03 04:06:55.983840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.983799 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Nov 03 04:06:55.984029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.983876 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Nov 03 04:06:55.993793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.993747 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\" (UniqueName: \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\") pod \"63d5bd27-9451-4b02-9491-f462d6d9ff10\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " Nov 03 04:06:55.994020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.993801 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\") pod \"63d5bd27-9451-4b02-9491-f462d6d9ff10\" (UID: \"63d5bd27-9451-4b02-9491-f462d6d9ff10\") " Nov 03 04:06:55.994237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.994215 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume" pod "63d5bd27-9451-4b02-9491-f462d6d9ff10" (UID: "63d5bd27-9451-4b02-9491-f462d6d9ff10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:55.994399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.994382 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d5bd27-9451-4b02-9491-f462d6d9ff10-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:55.996130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:55.996108 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4" pod "63d5bd27-9451-4b02-9491-f462d6d9ff10" (UID: "63d5bd27-9451-4b02-9491-f462d6d9ff10"). InnerVolumeSpecName "kube-api-access-g2nb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:56.001697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.001669 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 04:06:56.001822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.001731 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 04:06:56.095864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.095797 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nb4\" (UniqueName: \"kubernetes.io/projected/63d5bd27-9451-4b02-9491-f462d6d9ff10-kube-api-access-g2nb4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:56.378806 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.378771 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:06:56.385959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.385917 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-55dc0c-658bfb5f69-czhkd"] Nov 03 04:06:56.992512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:56.992467 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d5bd27-9451-4b02-9491-f462d6d9ff10" path="/var/lib/kubelet/pods/63d5bd27-9451-4b02-9491-f462d6d9ff10/volumes" Nov 03 04:06:57.693367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.693332 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:57.694248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.694221 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:57.695007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.694985 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:06:57.858139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.858101 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:57.858551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.858529 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:06:57.859528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:06:57.859507 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" Nov 03 04:07:02.331789 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:07:02.331746 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:07:16.051343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.051309 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb"] Nov 03 04:07:16.101039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.101004 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw"] Nov 03 04:07:16.104369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.104340 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d5ee23b-pldwb"] Nov 03 04:07:16.107477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.107451 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e9d836-2gcmw"] Nov 03 04:07:16.558477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.558444 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc"] Nov 03 04:07:16.559188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.559161 5275 cpu_manager.go:414] "containerMap: removing container" podUID="63d5bd27-9451-4b02-9491-f462d6d9ff10" containerName="payload-processing" Nov 03 04:07:16.559188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.559180 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d5bd27-9451-4b02-9491-f462d6d9ff10" containerName="payload-processing" Nov 03 04:07:16.559383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.559322 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d5bd27-9451-4b02-9491-f462d6d9ff10" containerName="payload-processing" Nov 03 04:07:16.564128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.564107 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.616482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.616443 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:07:16.616672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.616646 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:07:16.616672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.616646 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:07:16.616799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.616734 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:07:16.620739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.620713 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc"] Nov 03 04:07:16.677226 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.677186 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.677416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.677331 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.677472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.677438 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmx66\" (UniqueName: \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.781764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.779585 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.781764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.779738 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.781764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.779913 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmx66\" (UniqueName: \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.781764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.781075 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.789228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.789196 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.841586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.841512 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmx66\" (UniqueName: \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.878223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.878183 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:16.987915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.987871 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4daa4c56-06bc-4a71-93dd-17b29c3af2b7" path="/var/lib/kubelet/pods/4daa4c56-06bc-4a71-93dd-17b29c3af2b7/volumes" Nov 03 04:07:16.988355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:16.988340 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e1224e-62ef-4cf5-9882-e3b9f2cd5abe" path="/var/lib/kubelet/pods/89e1224e-62ef-4cf5-9882-e3b9f2cd5abe/volumes" Nov 03 04:07:17.050327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:17.050288 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg"] Nov 03 04:07:17.053503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:17.053466 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3de38c-949pg"] Nov 03 04:07:17.260610 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:07:17.260577 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918e0f0e_6b5c_4740_af9b_c81a0dd3fcda.slice/crio-be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64 WatchSource:0}: Error finding container be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64: Status 404 returned error can't find the container with id be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64 Nov 03 04:07:17.261231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:17.261057 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc"] Nov 03 04:07:18.048867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.048804 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77"] Nov 03 04:07:18.098567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.098526 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8e545d-z6x77"] Nov 03 04:07:18.130910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.130871 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" event={"ID":"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda","Type":"ContainerStarted","Data":"d4e60c428ba8eea9d930479e88ce0ff007225f4673db6e44c415ca6eb884b187"} Nov 03 04:07:18.131096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.130936 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" event={"ID":"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda","Type":"ContainerStarted","Data":"be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64"} Nov 03 04:07:18.251300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.251239 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" podStartSLOduration=2.251218876 podStartE2EDuration="2.251218876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:18.246213946 +0000 UTC m=+3929.398902398" watchObservedRunningTime="2026-11-03 04:07:18.251218876 +0000 UTC m=+3929.403907299" Nov 03 04:07:18.985026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.984990 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be6ccaf-95fe-4749-8b75-84f21f9084fd" path="/var/lib/kubelet/pods/9be6ccaf-95fe-4749-8b75-84f21f9084fd/volumes" Nov 03 04:07:18.985595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:18.985573 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d687d4-b864-4e4f-913f-09d1a7ca7e32" path="/var/lib/kubelet/pods/a6d687d4-b864-4e4f-913f-09d1a7ca7e32/volumes" Nov 03 04:07:19.139670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:19.139637 5275 generic.go:356] "Generic (PLEG): container finished" podID="918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" containerID="d4e60c428ba8eea9d930479e88ce0ff007225f4673db6e44c415ca6eb884b187" exitCode=0 Nov 03 04:07:19.140071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:19.139722 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" event={"ID":"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda","Type":"ContainerDied","Data":"d4e60c428ba8eea9d930479e88ce0ff007225f4673db6e44c415ca6eb884b187"} Nov 03 04:07:20.324424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.324391 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:20.425966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.425886 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\") pod \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " Nov 03 04:07:20.426127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.425998 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\" (UniqueName: \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\") pod \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " Nov 03 04:07:20.426127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.426093 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\") pod \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\" (UID: \"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda\") " Nov 03 04:07:20.426330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.426302 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca" pod "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" (UID: "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:07:20.426471 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.426454 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:20.428324 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.428291 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66" pod "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" (UID: "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda"). InnerVolumeSpecName "kube-api-access-mmx66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:07:20.428324 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.428311 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca" pod "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" (UID: "918e0f0e-6b5c-4740-af9b-c81a0dd3fcda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:07:20.527490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.527444 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmx66\" (UniqueName: \"kubernetes.io/projected/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-kube-api-access-mmx66\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:20.527490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:20.527485 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:21.166619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:21.166579 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" Nov 03 04:07:21.166823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:21.166591 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc" event={"ID":"918e0f0e-6b5c-4740-af9b-c81a0dd3fcda","Type":"ContainerDied","Data":"be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64"} Nov 03 04:07:21.166823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:21.166768 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be12e281ed41b4458aebac02d15f5b6dc1924049ea10cd99d1d3e206ab31cb64" Nov 03 04:07:26.050108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.050065 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8"] Nov 03 04:07:26.100367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.100332 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5"] Nov 03 04:07:26.103431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.103409 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dbeda7-6hgq8"] Nov 03 04:07:26.106237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.106215 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd9602b-mk4b5"] Nov 03 04:07:26.997770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.997718 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1136b2-8585-493c-8934-64e55b3389dd" path="/var/lib/kubelet/pods/5d1136b2-8585-493c-8934-64e55b3389dd/volumes" Nov 03 04:07:26.998715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:26.998685 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80924389-892f-4e3a-95ad-3a03633f8bef" path="/var/lib/kubelet/pods/80924389-892f-4e3a-95ad-3a03633f8bef/volumes" Nov 03 04:07:27.050953 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:27.050911 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd"] Nov 03 04:07:27.101220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:27.101177 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b14b58-2z4jd"] Nov 03 04:07:28.988335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:28.988296 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa2010a-fed1-42f7-ad27-554fb3dc6e08" path="/var/lib/kubelet/pods/2fa2010a-fed1-42f7-ad27-554fb3dc6e08/volumes" Nov 03 04:07:29.075044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:29.075008 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm"] Nov 03 04:07:29.122372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:29.122336 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ce77ee-j74sm"] Nov 03 04:07:30.986642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:30.986610 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636edba7-4176-4caf-bb06-ce4a70acec15" path="/var/lib/kubelet/pods/636edba7-4176-4caf-bb06-ce4a70acec15/volumes" Nov 03 04:07:32.330919 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:07:32.330876 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:07:45.941640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.941600 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q"] Nov 03 04:07:45.942221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.942206 5275 cpu_manager.go:414] "containerMap: removing container" podUID="918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" containerName="revoke-keys" Nov 03 04:07:45.942273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.942222 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" containerName="revoke-keys" Nov 03 04:07:45.942357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.942348 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" containerName="revoke-keys" Nov 03 04:07:45.947390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.947367 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:45.998821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.998773 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94g79\" (UniqueName: \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:45.999048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.998904 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:45.999048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:45.999007 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.003135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.003102 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q"] Nov 03 04:07:46.049140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.049106 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:07:46.049140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.049139 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:07:46.049395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.049199 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:07:46.049395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.049291 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:07:46.100347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.100309 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.100549 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.100407 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.100549 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.100542 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94g79\" (UniqueName: \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.101180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.101155 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.102984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.102958 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.163087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.163050 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94g79\" (UniqueName: \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\") pod \"maas-api-revoke-sub-46e4b913-nz59q\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.261608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.261519 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:46.646391 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:46.646360 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q"] Nov 03 04:07:46.648187 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:07:46.648160 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf89ee6_a59e_48e6_a186_c2bfaf6c011f.slice/crio-85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a WatchSource:0}: Error finding container 85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a: Status 404 returned error can't find the container with id 85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a Nov 03 04:07:47.404822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:47.404784 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" event={"ID":"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f","Type":"ContainerStarted","Data":"c77500aa970d11c6963a195e017ef9f7494070be08bfd3302a950f2fa59bb130"} Nov 03 04:07:47.405338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:47.404842 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" event={"ID":"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f","Type":"ContainerStarted","Data":"85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a"} Nov 03 04:07:47.522980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:47.522925 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" podStartSLOduration=2.522912124 podStartE2EDuration="2.522912124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:47.520475909 +0000 UTC m=+3958.673164331" watchObservedRunningTime="2026-11-03 04:07:47.522912124 +0000 UTC m=+3958.675600544" Nov 03 04:07:48.133428 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.133391 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2"] Nov 03 04:07:48.137544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.137525 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230421 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230458 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230637 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230751 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnggg\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-kube-api-access-tnggg\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230791 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230906 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.230979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.230964 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.231015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.231003 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.231133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.231111 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.332887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.332814 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.332934 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnggg\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-kube-api-access-tnggg\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.332958 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333021 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333055 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333118 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333171 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333283 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333294 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333454 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333457 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.333646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.333612 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.334270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.334208 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.334703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.334679 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.336896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.336829 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.403936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.384478 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.403936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.401443 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2"] Nov 03 04:07:48.403936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.403411 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.404412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.404387 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnggg\" (UniqueName: \"kubernetes.io/projected/c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb-kube-api-access-tnggg\") pod \"maas-default-gateway-openshift-default-76f4876667-gc8n2\" (UID: \"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.428260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.428218 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" event={"ID":"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f","Type":"ContainerDied","Data":"c77500aa970d11c6963a195e017ef9f7494070be08bfd3302a950f2fa59bb130"} Nov 03 04:07:48.428723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.428096 5275 generic.go:356] "Generic (PLEG): container finished" podID="cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" containerID="c77500aa970d11c6963a195e017ef9f7494070be08bfd3302a950f2fa59bb130" exitCode=0 Nov 03 04:07:48.451762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.451731 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:48.893570 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:07:48.893539 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ee6ca5_6b92_40cd_8d22_e90006a2e0bb.slice/crio-9162b7226946c6ad3f43f5c3fcea33cda699c2e3c349f2f912615c6fb8c453d7 WatchSource:0}: Error finding container 9162b7226946c6ad3f43f5c3fcea33cda699c2e3c349f2f912615c6fb8c453d7: Status 404 returned error can't find the container with id 9162b7226946c6ad3f43f5c3fcea33cda699c2e3c349f2f912615c6fb8c453d7 Nov 03 04:07:48.894157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.894133 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2"] Nov 03 04:07:48.897040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.897006 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:07:48.897161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.897073 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:07:48.897161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:48.897107 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:07:49.438068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.438027 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" event={"ID":"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb","Type":"ContainerStarted","Data":"49237962af2cd95a10539f1a4cbf8a406ba2dea9bef6090bfaaa464d9a2438a3"} Nov 03 04:07:49.438498 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.438094 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" event={"ID":"c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb","Type":"ContainerStarted","Data":"9162b7226946c6ad3f43f5c3fcea33cda699c2e3c349f2f912615c6fb8c453d7"} Nov 03 04:07:49.452050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.452018 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:49.453914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.453882 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:49.454052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.453942 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:49.560490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.560433 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podStartSLOduration=1.560416804 podStartE2EDuration="1.560416804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:49.556920486 +0000 UTC m=+3960.709608910" watchObservedRunningTime="2026-11-03 04:07:49.560416804 +0000 UTC m=+3960.713105225" Nov 03 04:07:49.601975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.601946 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:49.650993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.650954 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\") pod \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " Nov 03 04:07:49.650993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.651040 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\") pod \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " Nov 03 04:07:49.651462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.651083 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\" (UniqueName: \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\") pod \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\" (UID: \"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f\") " Nov 03 04:07:49.651462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.651348 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca" pod "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" (UID: "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:07:49.651743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.651714 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:49.653792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.653749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca" pod "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" (UID: "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:07:49.654004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.653982 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79" pod "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" (UID: "cbf89ee6-a59e-48e6-a186-c2bfaf6c011f"). InnerVolumeSpecName "kube-api-access-94g79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:07:49.753296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.753133 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:49.753296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:49.753171 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94g79\" (UniqueName: \"kubernetes.io/projected/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f-kube-api-access-94g79\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:50.446793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:50.446766 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" Nov 03 04:07:50.447328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:50.446806 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q" event={"ID":"cbf89ee6-a59e-48e6-a186-c2bfaf6c011f","Type":"ContainerDied","Data":"85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a"} Nov 03 04:07:50.447328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:50.446869 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85182437a7f32543a68a4b4cceebcb73a7601c147fb8845836de784d9115c17a" Nov 03 04:07:50.472874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:50.454238 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:50.472874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:50.454293 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:51.453321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:51.453283 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:51.453986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:51.453949 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:52.456933 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:52.453256 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:52.456933 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:52.453318 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:53.455873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:53.454289 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:53.455873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:53.454361 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:54.461761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:54.461297 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:54.461761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:54.461358 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:55.452880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:55.452830 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:55.453075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:55.452898 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:56.452858 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:56.452813 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 03 04:07:56.453302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:56.452892 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 03 04:07:58.034088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:58.034048 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:58.034564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:58.034542 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:58.035777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:58.035751 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" Nov 03 04:07:58.338286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:58.338193 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:07:59.612264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:59.612227 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:07:59.612892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:07:59.612458 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" containerID="cri-o://339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675" gracePeriod=30 Nov 03 04:08:02.331792 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:02.331749 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:08:03.064539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.064452 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:08:03.065288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.065264 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" containerName="revoke-keys" Nov 03 04:08:03.065288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.065286 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" containerName="revoke-keys" Nov 03 04:08:03.065503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.065400 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" containerName="revoke-keys" Nov 03 04:08:03.069046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.069027 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.138291 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.137932 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ed4ecc-openshift-default-dockercfg-ws2jw\"" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.149986 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150045 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150101 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150183 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150227 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150271 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150303 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150346 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6s9c\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.150919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.150377 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.237896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.237862 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:08:03.251868 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.251815 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.252156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.252134 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.252539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.252518 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.252790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.252772 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.253039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.253021 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.253203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.253185 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.253436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.253405 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.253611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.253596 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s9c\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.253731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.253718 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.254862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.254822 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.264875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.255669 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.274349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.255942 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.276020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.256076 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.283468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.256345 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.284431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.284392 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.286867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.286825 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.316987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.316914 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.319085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.319053 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s9c\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\") pod \"e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.339518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.339481 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" podUID="2b797ad9-d283-461c-ad0c-692d992c722c" containerName="payload-processing" containerID="cri-o://4154827367ea0e456d28b7a599f0f713de7c2fb0127e61d2482da38709328936" gracePeriod=25 Nov 03 04:08:03.382964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.382920 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:03.619519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.619485 5275 generic.go:356] "Generic (PLEG): container finished" podID="2b797ad9-d283-461c-ad0c-692d992c722c" containerID="4154827367ea0e456d28b7a599f0f713de7c2fb0127e61d2482da38709328936" exitCode=0 Nov 03 04:08:03.619758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.619535 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" event={"ID":"2b797ad9-d283-461c-ad0c-692d992c722c","Type":"ContainerDied","Data":"4154827367ea0e456d28b7a599f0f713de7c2fb0127e61d2482da38709328936"} Nov 03 04:08:03.674545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.674510 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:08:03.762390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.762346 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\") pod \"2b797ad9-d283-461c-ad0c-692d992c722c\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " Nov 03 04:08:03.762608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.762579 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\") pod \"2b797ad9-d283-461c-ad0c-692d992c722c\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " Nov 03 04:08:03.762681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.762645 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\") pod \"2b797ad9-d283-461c-ad0c-692d992c722c\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " Nov 03 04:08:03.762746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.762706 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\") pod \"2b797ad9-d283-461c-ad0c-692d992c722c\" (UID: \"2b797ad9-d283-461c-ad0c-692d992c722c\") " Nov 03 04:08:03.762825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.762797 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume" pod "2b797ad9-d283-461c-ad0c-692d992c722c" (UID: "2b797ad9-d283-461c-ad0c-692d992c722c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:03.763073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.763049 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume" pod "2b797ad9-d283-461c-ad0c-692d992c722c" (UID: "2b797ad9-d283-461c-ad0c-692d992c722c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:03.763350 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.763332 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:03.763500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.763353 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b797ad9-d283-461c-ad0c-692d992c722c-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:03.765150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.765123 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq" pod "2b797ad9-d283-461c-ad0c-692d992c722c" (UID: "2b797ad9-d283-461c-ad0c-692d992c722c"). InnerVolumeSpecName "kube-api-access-p7bkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:03.765423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.765405 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials" pod "2b797ad9-d283-461c-ad0c-692d992c722c" (UID: "2b797ad9-d283-461c-ad0c-692d992c722c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:03.863828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.863794 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:08:03.864692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.864603 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:03.864692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.864638 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7bkq\" (UniqueName: \"kubernetes.io/projected/2b797ad9-d283-461c-ad0c-692d992c722c-kube-api-access-p7bkq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:03.866822 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:03.866792 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfded98_72c8_4567_b05c_e5a1386f8f1a.slice/crio-ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5 WatchSource:0}: Error finding container ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5: Status 404 returned error can't find the container with id ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5 Nov 03 04:08:03.869707 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.869668 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:03.869816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.869763 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:03.869907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:03.869816 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:04.493740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.493689 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:08:04.494526 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.494492 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" containerID="cri-o://b8a222380361996d2831931b7f9f3d522e35d34831c0c0606ac85ce93e44ebd8" gracePeriod=30 Nov 03 04:08:04.646215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.646177 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" event={"ID":"2b797ad9-d283-461c-ad0c-692d992c722c","Type":"ContainerDied","Data":"e0b2cd7a93daff526c9d0d507cac52a8ed630571c3120713b9e957872d9f3347"} Nov 03 04:08:04.646374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.646260 5275 scope.go:122] "RemoveContainer" containerID="4154827367ea0e456d28b7a599f0f713de7c2fb0127e61d2482da38709328936" Nov 03 04:08:04.646467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.646450 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw" Nov 03 04:08:04.692868 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.687809 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerStarted","Data":"f5a3da2b1b42a04526626524c0151ff0b60fd694610ac9bb6fedb3c07268203f"} Nov 03 04:08:04.692868 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.687919 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerStarted","Data":"ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5"} Nov 03 04:08:04.824918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.821634 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podStartSLOduration=1.821618169 podStartE2EDuration="1.821618169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:04.821013171 +0000 UTC m=+3975.973701584" watchObservedRunningTime="2026-11-03 04:08:04.821618169 +0000 UTC m=+3975.974306589" Nov 03 04:08:04.898188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.898119 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:08:04.935264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.935223 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:08:04.940681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.940646 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c9b799cc-6jgjw"] Nov 03 04:08:04.991005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:04.990975 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b797ad9-d283-461c-ad0c-692d992c722c" path="/var/lib/kubelet/pods/2b797ad9-d283-461c-ad0c-692d992c722c/volumes" Nov 03 04:08:05.000239 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000212 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000352 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000426 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000487 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000503 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:05.000699 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000598 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000699 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000673 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000699 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000675 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:05.000886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000714 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000752 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.000886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000816 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\") pod \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\" (UID: \"ad62fb63-8fe6-49d5-b3a9-86da11f61230\") " Nov 03 04:08:05.001061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.000974 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:05.001271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001240 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001324 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001457 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001475 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001490 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001503 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.001832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.001518 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.003237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.003212 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:05.003447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.003419 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:08:05.003447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.003430 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:05.003580 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.003505 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p" pod "ad62fb63-8fe6-49d5-b3a9-86da11f61230" (UID: "ad62fb63-8fe6-49d5-b3a9-86da11f61230"). InnerVolumeSpecName "kube-api-access-r9z6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:05.103344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.103151 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9z6p\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-kube-api-access-r9z6p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.103344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.103192 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.103344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.103207 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.103344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.103221 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad62fb63-8fe6-49d5-b3a9-86da11f61230-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:05.383947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.383899 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:05.390404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.390377 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:05.698484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.698387 5275 generic.go:356] "Generic (PLEG): container finished" podID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerID="339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675" exitCode=0 Nov 03 04:08:05.698484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.698441 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerDied","Data":"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675"} Nov 03 04:08:05.699005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.698510 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" event={"ID":"ad62fb63-8fe6-49d5-b3a9-86da11f61230","Type":"ContainerDied","Data":"96f8b74f105c4c84adcfa750db3e7e44586859dcc2ddf0d55f464a29ee6c5710"} Nov 03 04:08:05.699005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.698536 5275 scope.go:122] "RemoveContainer" containerID="339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675" Nov 03 04:08:05.699005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.698449 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86" Nov 03 04:08:05.699158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.699006 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:05.700524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.699932 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:08:05.711804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.711778 5275 scope.go:122] "RemoveContainer" containerID="79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f" Nov 03 04:08:05.722631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.722604 5275 scope.go:122] "RemoveContainer" containerID="339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675" Nov 03 04:08:05.723112 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:05.723076 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675\": container with ID starting with 339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675 not found: ID does not exist" containerID="339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675" Nov 03 04:08:05.723183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.723122 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675"} err="failed to get container status \"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675\": rpc error: code = NotFound desc = could not find container \"339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675\": container with ID starting with 339fc8718a9d68c9b15a11d8760a8c22442cbbf0f978412d15613f59f1d2a675 not found: ID does not exist" Nov 03 04:08:05.723183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.723142 5275 scope.go:122] "RemoveContainer" containerID="79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f" Nov 03 04:08:05.723492 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:05.723407 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f\": container with ID starting with 79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f not found: ID does not exist" containerID="79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f" Nov 03 04:08:05.723492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.723436 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f"} err="failed to get container status \"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f\": rpc error: code = NotFound desc = could not find container \"79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f\": container with ID starting with 79fbb7894a9508457c2f6460702c59acf1b8f29191db6a3093d8482daa29e84f not found: ID does not exist" Nov 03 04:08:05.890874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.886404 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:08:05.890874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:05.886547 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-90139c-openshift-default-7949cf9c6f-bcw86"] Nov 03 04:08:06.040205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040173 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:06.040915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040892 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.040915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040914 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.041103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040936 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.041103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040943 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.041103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.040996 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2b797ad9-d283-461c-ad0c-692d992c722c" containerName="payload-processing" Nov 03 04:08:06.041103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.041004 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b797ad9-d283-461c-ad0c-692d992c722c" containerName="payload-processing" Nov 03 04:08:06.041313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.041187 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.041313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.041203 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" containerName="istio-proxy" Nov 03 04:08:06.041313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.041222 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b797ad9-d283-461c-ad0c-692d992c722c" containerName="payload-processing" Nov 03 04:08:06.046322 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.046300 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.098826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098785 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a116bf5ad870\"" Nov 03 04:08:06.098826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098785 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 04:08:06.098826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098811 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 04:08:06.098826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098825 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 04:08:06.099134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098920 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 04:08:06.099134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.098957 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4gwzc\"" Nov 03 04:08:06.100467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.100443 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:06.122998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.122957 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.122998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.122993 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9r5\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.123247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.123125 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.123247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.123198 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.153231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.153197 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:08:06.157456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.157430 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.210330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.210301 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-214eab-openshift-default-dockercfg-krqhm\"" Nov 03 04:08:06.218188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.218159 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:08:06.225317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.225282 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.225517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.225327 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9r5\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.225517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.225415 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.225517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.225468 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.226099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.226072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.226545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.226484 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.228455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.228431 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.287082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.287048 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9r5\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\") pod \"payload-processing-external-model-6b4c969f7c-zlmrh\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.333929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.333784 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.333929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.333893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334079 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334116 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl29\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334152 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334212 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334280 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334334 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.334510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.334365 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.377936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.377902 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:06.435416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435383 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435422 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435493 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435528 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435594 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435611 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl29\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435631 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435667 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.435901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.435706 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.436330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.436307 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.436713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.436678 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.436713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.436686 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.436890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.436713 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.437152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.437124 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.439644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.439621 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.440186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.440164 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.509183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.508143 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl29\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.510679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.510437 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\") pod \"e2e-inf-a-214eab-openshift-default-59849698df-zqrs6\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.773675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.773637 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:06.811234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.811200 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:06.820708 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:06.820678 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f462e27_39fc_48bd_b2ec_32aa3d3e2fe1.slice/crio-5589e42a3d6573413b0d7576258c75c34f013c2246e5ed7b24ddf8aedc5d8add WatchSource:0}: Error finding container 5589e42a3d6573413b0d7576258c75c34f013c2246e5ed7b24ddf8aedc5d8add: Status 404 returned error can't find the container with id 5589e42a3d6573413b0d7576258c75c34f013c2246e5ed7b24ddf8aedc5d8add Nov 03 04:08:06.986074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:06.986042 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad62fb63-8fe6-49d5-b3a9-86da11f61230" path="/var/lib/kubelet/pods/ad62fb63-8fe6-49d5-b3a9-86da11f61230/volumes" Nov 03 04:08:07.170737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.170702 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:08:07.171793 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:07.171764 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab67cc29_e51f_483e_9fde_3e60da64574b.slice/crio-183c2be180135f03b2ecece4b7535a466796423d00c7ea369ef050feecace17a WatchSource:0}: Error finding container 183c2be180135f03b2ecece4b7535a466796423d00c7ea369ef050feecace17a: Status 404 returned error can't find the container with id 183c2be180135f03b2ecece4b7535a466796423d00c7ea369ef050feecace17a Nov 03 04:08:07.174869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.174817 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:07.174986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.174936 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:07.175055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.174982 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:07.549669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.549636 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:08:07.550142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.550107 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" containerID="cri-o://8417279e265aa0b7fa1576e7d6e33402c08f2f58c0ec75949bde83e7af755ca3" gracePeriod=30 Nov 03 04:08:07.757820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.756583 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" event={"ID":"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1","Type":"ContainerStarted","Data":"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3"} Nov 03 04:08:07.757820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.756653 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" event={"ID":"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1","Type":"ContainerStarted","Data":"5589e42a3d6573413b0d7576258c75c34f013c2246e5ed7b24ddf8aedc5d8add"} Nov 03 04:08:07.757820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.757757 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:07.763016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.762984 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerStarted","Data":"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97"} Nov 03 04:08:07.763275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.763257 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerStarted","Data":"183c2be180135f03b2ecece4b7535a466796423d00c7ea369ef050feecace17a"} Nov 03 04:08:07.774450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.774418 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:07.862828 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:07.862742 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:07.863091 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:07.863066 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:07.863219 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:07.863153 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:08.363135784 +0000 UTC m=+3979.515824183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:07.876753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.876688 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" podStartSLOduration=2.876673382 podStartE2EDuration="2.876673382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:07.872779184 +0000 UTC m=+3979.025467616" watchObservedRunningTime="2026-11-03 04:08:07.876673382 +0000 UTC m=+3979.029361833" Nov 03 04:08:07.993817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:07.993727 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podStartSLOduration=1.9936997810000001 podStartE2EDuration="1.993699781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:07.99231991 +0000 UTC m=+3979.145008331" watchObservedRunningTime="2026-11-03 04:08:07.993699781 +0000 UTC m=+3979.146388251" Nov 03 04:08:08.370535 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:08.370502 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:08.370535 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:08.370528 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:08.370771 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:08.370579 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:09.370565259 +0000 UTC m=+3980.523253659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:08.780425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:08.780390 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:08.894804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:08.894768 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:09.385871 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:09.385819 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:09.385871 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:09.385862 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:09.386069 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:09.385931 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:11.385911675 +0000 UTC m=+3982.538600079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:09.789883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.787936 5275 generic.go:356] "Generic (PLEG): container finished" podID="cc5db47a-df8b-4df0-8d97-112092993e04" containerID="b8a222380361996d2831931b7f9f3d522e35d34831c0c0606ac85ce93e44ebd8" exitCode=0 Nov 03 04:08:09.789883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.789371 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerDied","Data":"b8a222380361996d2831931b7f9f3d522e35d34831c0c0606ac85ce93e44ebd8"} Nov 03 04:08:09.789883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.789437 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:09.789883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.789466 5275 scope.go:122] "RemoveContainer" containerID="2b32c04d53f6f345aa3b537021856a40cd06320e7be7c3cabbd7e60a949eb192" Nov 03 04:08:09.789883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.789812 5275 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" secret="" err="secret \"payload-processing-external-model-dockercfg-4gwzc\" not found" Nov 03 04:08:09.798875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:09.792573 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:08:09.894023 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:09.893978 5275 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:09.894239 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:09.894083 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:10.39406316 +0000 UTC m=+3981.546751578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:10.271918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.271891 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:08:10.401913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.401794 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.401963 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402015 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402101 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402133 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402189 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402249 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402290 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402318 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\") pod \"cc5db47a-df8b-4df0-8d97-112092993e04\" (UID: \"cc5db47a-df8b-4df0-8d97-112092993e04\") " Nov 03 04:08:10.402403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402376 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:10.402603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402563 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:10.403018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402837 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:10.403018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402936 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.403018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.402955 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.403018 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:10.403019 5275 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:10.403349 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:10.403087 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:11.403065732 +0000 UTC m=+3982.555754137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:10.404013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.403921 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:10.404215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.404185 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:10.404558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.404537 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:10.404676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.404612 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:08:10.405962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.405932 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:10.407001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.406974 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9" pod "cc5db47a-df8b-4df0-8d97-112092993e04" (UID: "cc5db47a-df8b-4df0-8d97-112092993e04"). InnerVolumeSpecName "kube-api-access-wbdw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:10.504515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504479 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504516 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbdw9\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-kube-api-access-wbdw9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504534 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5db47a-df8b-4df0-8d97-112092993e04-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504548 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504562 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5db47a-df8b-4df0-8d97-112092993e04-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504575 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5db47a-df8b-4df0-8d97-112092993e04-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.504706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.504589 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5db47a-df8b-4df0-8d97-112092993e04-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:10.523354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.523317 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:08:10.523962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.523947 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:10.524030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.523964 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:10.524113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.524103 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:10.527955 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.527933 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.588952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.588914 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:08:10.611281 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.611240 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c4fa79-openshift-default-dockercfg-gr4mc\"" Nov 03 04:08:10.706874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.706780 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.706874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.706821 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4js2\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.706883 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.706998 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.707048 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.707131 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.707199 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.707268 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.707404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.707392 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.799174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.799141 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" Nov 03 04:08:10.799712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.799179 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt" event={"ID":"cc5db47a-df8b-4df0-8d97-112092993e04","Type":"ContainerDied","Data":"921e0ce170abba32d20b3d2c514214e88a0731bd865493cee266324d74848620"} Nov 03 04:08:10.799712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.799235 5275 scope.go:122] "RemoveContainer" containerID="b8a222380361996d2831931b7f9f3d522e35d34831c0c0606ac85ce93e44ebd8" Nov 03 04:08:10.809027 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.808993 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809193 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809168 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809226 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809270 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4js2\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809316 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809386 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809430 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809524 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809619 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809760 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.809942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809916 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.810025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.809996 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.810144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.810096 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.810270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.810244 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.812126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.812098 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.813233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.813213 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.871715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.871667 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.871986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.871675 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4js2\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\") pod \"e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:10.965753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.965669 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:08:10.971043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.971014 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03da9e-openshift-default-75d57f84dc-xjnbt"] Nov 03 04:08:10.987080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:10.987044 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" path="/var/lib/kubelet/pods/cc5db47a-df8b-4df0-8d97-112092993e04/volumes" Nov 03 04:08:11.141725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.141690 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:11.425619 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:11.425583 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:11.425619 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:11.425605 5275 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:11.425619 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:11.425615 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:11.425941 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:11.425685 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:13.425665301 +0000 UTC m=+3984.578353715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:11.425941 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:11.425707 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:15.425697256 +0000 UTC m=+3986.578385683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:11.584898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.584808 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:08:11.585390 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:11.585366 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c345e5_b98b_4631_ac99_bafe79cccb05.slice/crio-ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb WatchSource:0}: Error finding container ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb: Status 404 returned error can't find the container with id ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb Nov 03 04:08:11.588302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.588269 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:11.588385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.588338 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:11.588385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.588367 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:11.809871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.809277 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerStarted","Data":"83d453970edaa910f9f1a96550e50acbb2db0baa2586e4e61e8c340e42321963"} Nov 03 04:08:11.809871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.809333 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerStarted","Data":"ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb"} Nov 03 04:08:11.933086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:11.932955 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podStartSLOduration=1.932936064 podStartE2EDuration="1.932936064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:11.929704655 +0000 UTC m=+3983.082393076" watchObservedRunningTime="2026-11-03 04:08:11.932936064 +0000 UTC m=+3983.085624486" Nov 03 04:08:12.141919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.141877 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:12.147982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.147955 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:12.821278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.821210 5275 generic.go:356] "Generic (PLEG): container finished" podID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerID="8417279e265aa0b7fa1576e7d6e33402c08f2f58c0ec75949bde83e7af755ca3" exitCode=0 Nov 03 04:08:12.821665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.821280 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerDied","Data":"8417279e265aa0b7fa1576e7d6e33402c08f2f58c0ec75949bde83e7af755ca3"} Nov 03 04:08:12.821665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.821341 5275 scope.go:122] "RemoveContainer" containerID="69c6f4e24eb30efa8a8231d5561c8a7b16a34218319d958f9bcff9402ccea24e" Nov 03 04:08:12.821665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.821539 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:12.822538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.822511 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:08:12.847423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.847402 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:08:12.952694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.952654 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.952694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.952704 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.952751 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.952829 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.952947 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953015 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953053 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:12.953183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953064 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953168 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:12.953410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953202 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953250 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\") pod \"ddbda912-9cf0-416c-99a2-5536aaee1db5\" (UID: \"ddbda912-9cf0-416c-99a2-5536aaee1db5\") " Nov 03 04:08:12.953410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953256 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:12.953410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953307 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:12.953645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953562 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:12.953937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953898 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:12.953937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953922 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:12.953937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953936 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:12.954168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953951 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbda912-9cf0-416c-99a2-5536aaee1db5-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:12.954168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.953964 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:12.955520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.955496 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:12.955638 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.955596 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "kube-api-access-gzt9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:12.955710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.955649 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:12.955824 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:12.955808 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo" pod "ddbda912-9cf0-416c-99a2-5536aaee1db5" (UID: "ddbda912-9cf0-416c-99a2-5536aaee1db5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:08:13.056095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.056054 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:13.056095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.056094 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:13.056307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.056108 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt9w\" (UniqueName: \"kubernetes.io/projected/ddbda912-9cf0-416c-99a2-5536aaee1db5-kube-api-access-gzt9w\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:13.056307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.056121 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbda912-9cf0-416c-99a2-5536aaee1db5-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:13.462053 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:13.462017 5275 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:13.462222 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:13.462091 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume podName:0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1 nodeName:}" failed. No retries permitted until 2026-11-03 04:08:17.462076924 +0000 UTC m=+3988.614765327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume") pod "payload-processing-external-model-6b4c969f7c-zlmrh" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1") : configmap "payload-processing-external-model-plugins" not found Nov 03 04:08:13.831564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.831526 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" Nov 03 04:08:13.831564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.831540 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb" event={"ID":"ddbda912-9cf0-416c-99a2-5536aaee1db5","Type":"ContainerDied","Data":"b547871cf3f7f961fb56301769c5e9a8e1ea4a2ebd817ff3b01822c303faf138"} Nov 03 04:08:13.832190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:13.831623 5275 scope.go:122] "RemoveContainer" containerID="8417279e265aa0b7fa1576e7d6e33402c08f2f58c0ec75949bde83e7af755ca3" Nov 03 04:08:14.100957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:14.100376 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:08:14.109871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:14.105786 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-55dc0c-openshift-default-77b4bbdfc9-4dzmb"] Nov 03 04:08:14.987977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:14.987939 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" path="/var/lib/kubelet/pods/ddbda912-9cf0-416c-99a2-5536aaee1db5/volumes" Nov 03 04:08:15.799872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.799585 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" podUID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" containerName="payload-processing" containerID="cri-o://8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3" gracePeriod=25 Nov 03 04:08:15.877671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.877635 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:08:15.878367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878349 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:15.878478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878370 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:15.878478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878391 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:15.878478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878397 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:15.878652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878549 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:15.878652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878561 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5db47a-df8b-4df0-8d97-112092993e04" containerName="istio-proxy" Nov 03 04:08:15.878652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.878574 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:15.882115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.882094 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:15.934903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.934873 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:15.935163 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:15.935142 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6tz\" (UniqueName: \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.012877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.010840 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ed4ecc\"" Nov 03 04:08:16.012877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.010974 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ed4ecc-dockercfg-vctzh\"" Nov 03 04:08:16.028158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.017123 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:08:16.121193 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.119998 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:08:16.130882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.124526 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:16.130882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.124558 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbda912-9cf0-416c-99a2-5536aaee1db5" containerName="istio-proxy" Nov 03 04:08:16.130882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.125021 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.130882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.125282 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6tz\" (UniqueName: \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.130882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.126315 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.133833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.133808 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:08:16.134045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.133948 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.191180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.191035 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6tz\" (UniqueName: \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\") pod \"payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.221775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.221737 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.226884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.226814 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.227040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.227018 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxk5\" (UniqueName: \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.278924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.278891 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:16.328418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.328314 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\") pod \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " Nov 03 04:08:16.328613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.328456 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\") pod \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " Nov 03 04:08:16.328613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.328492 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\") pod \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " Nov 03 04:08:16.328613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.328536 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\") pod \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\" (UID: \"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1\") " Nov 03 04:08:16.329242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.328934 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume" pod "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:16.329242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329113 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.329242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329145 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume" pod "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:16.329242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329214 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxk5\" (UniqueName: \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.329553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329325 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:16.329553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329344 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:16.329953 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.329932 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.331229 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.331205 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials" pod "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:16.331346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.331324 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5" pod "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" (UID: "0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1"). InnerVolumeSpecName "kube-api-access-8p9r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:16.390606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.390516 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxk5\" (UniqueName: \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\") pod \"payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.430931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.430894 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:16.430931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.430932 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9r5\" (UniqueName: \"kubernetes.io/projected/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1-kube-api-access-8p9r5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:16.452670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.452634 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:16.626313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.626285 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:08:16.629464 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:16.629431 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740ab299_86ab_411f_a5b9_a3a94ee2da26.slice/crio-0424600cd2b0b1c46ec0a71c60c16a61c66d02e137556d16436eb85e944a48fe WatchSource:0}: Error finding container 0424600cd2b0b1c46ec0a71c60c16a61c66d02e137556d16436eb85e944a48fe: Status 404 returned error can't find the container with id 0424600cd2b0b1c46ec0a71c60c16a61c66d02e137556d16436eb85e944a48fe Nov 03 04:08:16.867961 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.867923 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" event={"ID":"740ab299-86ab-411f-a5b9-a3a94ee2da26","Type":"ContainerStarted","Data":"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a"} Nov 03 04:08:16.868178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.868028 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:16.868178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.868058 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" event={"ID":"740ab299-86ab-411f-a5b9-a3a94ee2da26","Type":"ContainerStarted","Data":"0424600cd2b0b1c46ec0a71c60c16a61c66d02e137556d16436eb85e944a48fe"} Nov 03 04:08:16.869638 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.869607 5275 generic.go:356] "Generic (PLEG): container finished" podID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" containerID="8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3" exitCode=0 Nov 03 04:08:16.869797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.869648 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" event={"ID":"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1","Type":"ContainerDied","Data":"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3"} Nov 03 04:08:16.869797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.869662 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" Nov 03 04:08:16.869797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.869695 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh" event={"ID":"0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1","Type":"ContainerDied","Data":"5589e42a3d6573413b0d7576258c75c34f013c2246e5ed7b24ddf8aedc5d8add"} Nov 03 04:08:16.869797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.869726 5275 scope.go:122] "RemoveContainer" containerID="8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3" Nov 03 04:08:16.878355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.878322 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:08:16.878652 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:16.878624 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf88a52_6914_45c8_bee4_80b7e85d4ecc.slice/crio-21bd35ba7603bd5366e63f4a69454d7d817edfd3c8f38b81f83124f1caf68a90 WatchSource:0}: Error finding container 21bd35ba7603bd5366e63f4a69454d7d817edfd3c8f38b81f83124f1caf68a90: Status 404 returned error can't find the container with id 21bd35ba7603bd5366e63f4a69454d7d817edfd3c8f38b81f83124f1caf68a90 Nov 03 04:08:16.883775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.883753 5275 scope.go:122] "RemoveContainer" containerID="8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3" Nov 03 04:08:16.884177 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:16.884152 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3\": container with ID starting with 8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3 not found: ID does not exist" containerID="8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3" Nov 03 04:08:16.884287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.884183 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3"} err="failed to get container status \"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3\": rpc error: code = NotFound desc = could not find container \"8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3\": container with ID starting with 8a55648807d6986dd3f4bb0230456d5fe4bbbdf0a09ec4c425026e1d225305d3 not found: ID does not exist" Nov 03 04:08:16.987526 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:16.987459 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" podStartSLOduration=1.987440211 podStartE2EDuration="1.987440211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:16.983197321 +0000 UTC m=+3988.135885742" watchObservedRunningTime="2026-11-03 04:08:16.987440211 +0000 UTC m=+3988.140128633" Nov 03 04:08:17.095644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.095547 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:17.102122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.102087 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-zlmrh"] Nov 03 04:08:17.879625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.879590 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" event={"ID":"6cf88a52-6914-45c8-bee4-80b7e85d4ecc","Type":"ContainerStarted","Data":"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e"} Nov 03 04:08:17.879873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.879655 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:17.879873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.879678 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" event={"ID":"6cf88a52-6914-45c8-bee4-80b7e85d4ecc","Type":"ContainerStarted","Data":"21bd35ba7603bd5366e63f4a69454d7d817edfd3c8f38b81f83124f1caf68a90"} Nov 03 04:08:17.996956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:17.995440 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" podStartSLOduration=2.995421265 podStartE2EDuration="2.995421265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:17.994657613 +0000 UTC m=+3989.147346036" watchObservedRunningTime="2026-11-03 04:08:17.995421265 +0000 UTC m=+3989.148109687" Nov 03 04:08:19.023870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.023584 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" path="/var/lib/kubelet/pods/0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1/volumes" Nov 03 04:08:19.179064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.179023 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:19.180006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.179986 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" containerName="payload-processing" Nov 03 04:08:19.180058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.180009 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" containerName="payload-processing" Nov 03 04:08:19.180218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.180206 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f462e27-39fc-48bd-b2ec-32aa3d3e2fe1" containerName="payload-processing" Nov 03 04:08:19.184505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.184483 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.238924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.238888 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:19.282000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.281893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.282000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.281995 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24lzm\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.282270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.282244 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.282363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.282296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.284311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.284287 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 04:08:19.284311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.284296 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a116bf5ad870\"" Nov 03 04:08:19.284517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.284335 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 04:08:19.284517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.284387 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 04:08:19.284517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.284402 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 04:08:19.286192 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.286171 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kxphp\"" Nov 03 04:08:19.383537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.383494 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24lzm\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.383915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.383688 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.383915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.383727 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.383915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.383775 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.384652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.384625 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.384793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.384687 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.386806 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.386780 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.444990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.444953 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24lzm\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\") pod \"payload-processing-external-model-6b4c969f7c-wjq85\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.500325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.500284 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:19.884445 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:19.883279 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e791b9_b21f_47ec_a85a_ec8b626d85ce.slice/crio-4414bea94d4134122b881a8947f02dc36b0c44a94b755b70370d60fb0e935b33 WatchSource:0}: Error finding container 4414bea94d4134122b881a8947f02dc36b0c44a94b755b70370d60fb0e935b33: Status 404 returned error can't find the container with id 4414bea94d4134122b881a8947f02dc36b0c44a94b755b70370d60fb0e935b33 Nov 03 04:08:19.884688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.884582 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:19.898446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:19.898413 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" event={"ID":"77e791b9-b21f-47ec-a85a-ec8b626d85ce","Type":"ContainerStarted","Data":"4414bea94d4134122b881a8947f02dc36b0c44a94b755b70370d60fb0e935b33"} Nov 03 04:08:20.907148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:20.907111 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" event={"ID":"77e791b9-b21f-47ec-a85a-ec8b626d85ce","Type":"ContainerStarted","Data":"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032"} Nov 03 04:08:20.907643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:20.907211 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:21.018157 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.018119 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:21.018157 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.018150 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:21.018394 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.018214 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials podName:77e791b9-b21f-47ec-a85a-ec8b626d85ce nodeName:}" failed. No retries permitted until 2026-11-03 04:08:21.518192833 +0000 UTC m=+3992.670881232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-wjq85" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:21.024392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:21.024334 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" podStartSLOduration=2.024320883 podStartE2EDuration="2.024320883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:21.023210389 +0000 UTC m=+3992.175898811" watchObservedRunningTime="2026-11-03 04:08:21.024320883 +0000 UTC m=+3992.177009304" Nov 03 04:08:21.524520 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.524489 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:21.524520 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.524518 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:21.524742 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:21.524576 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials podName:77e791b9-b21f-47ec-a85a-ec8b626d85ce nodeName:}" failed. No retries permitted until 2026-11-03 04:08:22.524559749 +0000 UTC m=+3993.677248161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-wjq85" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:22.538664 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:22.538627 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:22.538664 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:22.538661 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:22.539194 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:22.538738 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials podName:77e791b9-b21f-47ec-a85a-ec8b626d85ce nodeName:}" failed. No retries permitted until 2026-11-03 04:08:24.538714813 +0000 UTC m=+3995.691403228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-wjq85" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:23.434097 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.434065 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:08:23.439651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.439630 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.480134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.478192 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmchh\" (UniqueName: \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.480134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.478334 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.492247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.492211 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-214eab\"" Nov 03 04:08:23.497049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.497008 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:08:23.541834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.541801 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-214eab-dockercfg-ndc4q\"" Nov 03 04:08:23.594263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.584887 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmchh\" (UniqueName: \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.594263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.585032 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.594263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.586052 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.632340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.628395 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:08:23.633931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.633902 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.671580 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.660338 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:08:23.679709 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.679642 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmchh\" (UniqueName: \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\") pod \"payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.689922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.686632 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.689922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.686738 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptz8l\" (UniqueName: \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.755109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.755069 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:23.788906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.788833 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.789092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.788979 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptz8l\" (UniqueName: \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.789613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.789587 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.850935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.850824 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptz8l\" (UniqueName: \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\") pod \"payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:23.989841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:23.989629 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:24.160429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.160285 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:08:24.407092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.406418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:08:24.413097 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:24.413056 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c358ccf_6162_45f8_aad6_3113d9b673d3.slice/crio-35d5c1b66036fcfe64eef9170b63f6795dfcd154067a9b1abc8aa672a1e1e4b8 WatchSource:0}: Error finding container 35d5c1b66036fcfe64eef9170b63f6795dfcd154067a9b1abc8aa672a1e1e4b8: Status 404 returned error can't find the container with id 35d5c1b66036fcfe64eef9170b63f6795dfcd154067a9b1abc8aa672a1e1e4b8 Nov 03 04:08:24.615697 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:24.615658 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:24.615697 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:24.615692 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:24.616332 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:24.615898 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials podName:77e791b9-b21f-47ec-a85a-ec8b626d85ce nodeName:}" failed. No retries permitted until 2026-11-03 04:08:28.615824611 +0000 UTC m=+3999.768513024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-wjq85" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:24.965823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.965782 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" event={"ID":"2c358ccf-6162-45f8-aad6-3113d9b673d3","Type":"ContainerStarted","Data":"24542c35e00d34c7c4d496d5e4394ab17a5323194c1f09e03d5411f630ec3d2d"} Nov 03 04:08:24.966204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.966178 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" event={"ID":"2c358ccf-6162-45f8-aad6-3113d9b673d3","Type":"ContainerStarted","Data":"35d5c1b66036fcfe64eef9170b63f6795dfcd154067a9b1abc8aa672a1e1e4b8"} Nov 03 04:08:24.966394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.966378 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:24.967644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.967617 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" event={"ID":"d5807b36-d606-45ba-a992-3bfff331e5d3","Type":"ContainerStarted","Data":"a7ab99f18c1c23c2f6acf7f5a82bc7c7f4c32855db813f49a4e8e291e82af980"} Nov 03 04:08:24.967780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.967659 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" event={"ID":"d5807b36-d606-45ba-a992-3bfff331e5d3","Type":"ContainerStarted","Data":"eb058e38b599b9b80d9821651f771829492ab117437585d018385a13d5aac7e1"} Nov 03 04:08:24.967780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:24.967709 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:25.113887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:25.101804 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" podStartSLOduration=2.101783496 podStartE2EDuration="2.101783496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:25.096333683 +0000 UTC m=+3996.249022105" watchObservedRunningTime="2026-11-03 04:08:25.101783496 +0000 UTC m=+3996.254471918" Nov 03 04:08:25.231935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:25.231503 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" podStartSLOduration=2.231481692 podStartE2EDuration="2.231481692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:25.217247597 +0000 UTC m=+3996.369936019" watchObservedRunningTime="2026-11-03 04:08:25.231481692 +0000 UTC m=+3996.384170149" Nov 03 04:08:27.883022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:27.882985 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:08:28.358733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:28.358696 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:08:28.358994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:28.358944 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" containerID="cri-o://c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796" gracePeriod=30 Nov 03 04:08:28.638036 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:28.637946 5275 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a116bf5ad870: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:28.638036 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:28.637982 5275 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85: secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:28.638238 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:28.638050 5275 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials podName:77e791b9-b21f-47ec-a85a-ec8b626d85ce nodeName:}" failed. No retries permitted until 2026-11-03 04:08:36.63802882 +0000 UTC m=+4007.790717241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials") pod "payload-processing-external-model-6b4c969f7c-wjq85" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce") : secret "agc-cred-a116bf5ad870" not found Nov 03 04:08:28.892379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:28.892294 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:08:29.214574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:29.214483 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:29.218007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:29.217981 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:32.330785 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:32.330751 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:08:33.066489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.066457 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc"] Nov 03 04:08:33.114742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.114708 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c832d-wz7wc"] Nov 03 04:08:33.283873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.282453 5275 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:08:33.283873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.282528 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:08:33.354273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.354186 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:08:33.355188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.355139 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" containerID="cri-o://af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd" gracePeriod=30 Nov 03 04:08:33.480016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.479981 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:08:33.496610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.491623 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.606951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.604228 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:08:33.643228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.643200 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 04:08:33.643228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.643212 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 04:08:33.643500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.643267 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 04:08:33.643500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.643327 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4d77v\"" Nov 03 04:08:33.670132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670094 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670198 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhml\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670275 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670303 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670335 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670403 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.670702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.670489 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.765523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.765496 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:08:33.771612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771576 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.771791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771622 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.771791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771669 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.771791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771757 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.771990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771834 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:33.771990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771878 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.772108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.771987 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:33.772108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772003 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.772108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772035 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.772108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772071 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:33.772108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772083 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.772383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772300 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.772383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772334 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.772383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772364 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.772555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772384 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:33.772555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772392 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:33.772555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772460 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772770 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.772976 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773080 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhml\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773193 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773210 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773224 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773238 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.773251 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.774590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.774579 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.775400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.774682 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:33.775400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.775136 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:08:33.775521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.775423 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.775597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.775559 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "kube-api-access-rqzdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:33.775798 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.775775 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.776183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.776160 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.776946 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.776925 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.832945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.832910 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.833185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.833169 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhml\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5t4rh\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.874199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.874099 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\") pod \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\" (UID: \"4aa17b61-8bab-4fa0-a556-6dd919a79ec2\") " Nov 03 04:08:33.874599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.874583 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.874681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.874606 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.874681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.874620 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzdx\" (UniqueName: \"kubernetes.io/projected/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-kube-api-access-rqzdx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:33.876369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.876344 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy" pod "4aa17b61-8bab-4fa0-a556-6dd919a79ec2" (UID: "4aa17b61-8bab-4fa0-a556-6dd919a79ec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:33.885542 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.885514 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:33.976023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:33.975989 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa17b61-8bab-4fa0-a556-6dd919a79ec2-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:34.154001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.153914 5275 generic.go:356] "Generic (PLEG): container finished" podID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerID="c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796" exitCode=0 Nov 03 04:08:34.154169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.154066 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" event={"ID":"4aa17b61-8bab-4fa0-a556-6dd919a79ec2","Type":"ContainerDied","Data":"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796"} Nov 03 04:08:34.154169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.154105 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" event={"ID":"4aa17b61-8bab-4fa0-a556-6dd919a79ec2","Type":"ContainerDied","Data":"b2f5b067fa92e92f213f5077e9ed9dadd64375abe7eecf6e8cd4a3d9407ed034"} Nov 03 04:08:34.154169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.154128 5275 scope.go:122] "RemoveContainer" containerID="c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796" Nov 03 04:08:34.154347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.154260 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26" Nov 03 04:08:34.171511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.171482 5275 scope.go:122] "RemoveContainer" containerID="c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796" Nov 03 04:08:34.171900 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:34.171878 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796\": container with ID starting with c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796 not found: ID does not exist" containerID="c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796" Nov 03 04:08:34.172000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.171909 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796"} err="failed to get container status \"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796\": rpc error: code = NotFound desc = could not find container \"c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796\": container with ID starting with c59d74cb5caef32d27cca7bb4d8ccfc37f65e5d6f05d2145305b169cf5e3f796 not found: ID does not exist" Nov 03 04:08:34.215928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.215883 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" podUID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" containerName="payload-processing" containerID="cri-o://bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032" gracePeriod=25 Nov 03 04:08:34.322533 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.322497 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:08:34.326884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.326831 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-bwk26"] Nov 03 04:08:34.371999 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:34.371962 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a277337_efd5_4e77_9ffa_11da1a86461a.slice/crio-8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c WatchSource:0}: Error finding container 8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c: Status 404 returned error can't find the container with id 8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c Nov 03 04:08:34.372840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.372812 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:08:34.375182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.375144 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:34.375278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.375223 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:08:34.513291 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.513266 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:34.583632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.583592 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\") pod \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " Nov 03 04:08:34.583825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.583680 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\") pod \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " Nov 03 04:08:34.583825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.583708 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\") pod \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " Nov 03 04:08:34.583825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.583726 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\") pod \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\" (UID: \"77e791b9-b21f-47ec-a85a-ec8b626d85ce\") " Nov 03 04:08:34.584163 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.584140 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume" pod "77e791b9-b21f-47ec-a85a-ec8b626d85ce" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:34.584251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.584220 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume" pod "77e791b9-b21f-47ec-a85a-ec8b626d85ce" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:34.584320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.584254 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:34.586166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.586140 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm" pod "77e791b9-b21f-47ec-a85a-ec8b626d85ce" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce"). InnerVolumeSpecName "kube-api-access-24lzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:34.586289 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.586173 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials" pod "77e791b9-b21f-47ec-a85a-ec8b626d85ce" (UID: "77e791b9-b21f-47ec-a85a-ec8b626d85ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:34.686111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.686074 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:34.686307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.686117 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e791b9-b21f-47ec-a85a-ec8b626d85ce-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:34.686307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.686134 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24lzm\" (UniqueName: \"kubernetes.io/projected/77e791b9-b21f-47ec-a85a-ec8b626d85ce-kube-api-access-24lzm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:34.989237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.989198 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" path="/var/lib/kubelet/pods/4aa17b61-8bab-4fa0-a556-6dd919a79ec2/volumes" Nov 03 04:08:34.989906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:34.989865 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918e0f0e-6b5c-4740-af9b-c81a0dd3fcda" path="/var/lib/kubelet/pods/918e0f0e-6b5c-4740-af9b-c81a0dd3fcda/volumes" Nov 03 04:08:35.164537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.164493 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" event={"ID":"0a277337-efd5-4e77-9ffa-11da1a86461a","Type":"ContainerStarted","Data":"1a33b14b3e226c7bc93e89e6a797f23fe984df0bbdb26d296f66e22acb5bd49d"} Nov 03 04:08:35.164773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.164556 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" event={"ID":"0a277337-efd5-4e77-9ffa-11da1a86461a","Type":"ContainerStarted","Data":"8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c"} Nov 03 04:08:35.164773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.164581 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:35.167207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.167178 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5t4rh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:08:35.167415 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.167229 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:08:35.169072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.168937 5275 generic.go:356] "Generic (PLEG): container finished" podID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" containerID="bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032" exitCode=0 Nov 03 04:08:35.169072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.168990 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" Nov 03 04:08:35.169256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.168983 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" event={"ID":"77e791b9-b21f-47ec-a85a-ec8b626d85ce","Type":"ContainerDied","Data":"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032"} Nov 03 04:08:35.169256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.169148 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85" event={"ID":"77e791b9-b21f-47ec-a85a-ec8b626d85ce","Type":"ContainerDied","Data":"4414bea94d4134122b881a8947f02dc36b0c44a94b755b70370d60fb0e935b33"} Nov 03 04:08:35.169256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.169202 5275 scope.go:122] "RemoveContainer" containerID="bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032" Nov 03 04:08:35.190557 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.190502 5275 scope.go:122] "RemoveContainer" containerID="bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032" Nov 03 04:08:35.191742 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:35.191710 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032\": container with ID starting with bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032 not found: ID does not exist" containerID="bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032" Nov 03 04:08:35.191885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.191749 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032"} err="failed to get container status \"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032\": rpc error: code = NotFound desc = could not find container \"bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032\": container with ID starting with bb1688e475b6b9909628b547cafabe20ac3705f2460ac7b22f7d334160078032 not found: ID does not exist" Nov 03 04:08:35.286918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.286822 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" podStartSLOduration=2.286806705 podStartE2EDuration="2.286806705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:35.284601645 +0000 UTC m=+4006.437290068" watchObservedRunningTime="2026-11-03 04:08:35.286806705 +0000 UTC m=+4006.439495126" Nov 03 04:08:35.397637 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.397601 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:35.405944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:35.405916 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4c969f7c-wjq85"] Nov 03 04:08:36.018876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:36.018821 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:08:36.019091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:36.019072 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:08:36.192775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:36.192744 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5t4rh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:08:36.193011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:36.192793 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:08:36.987212 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:36.987175 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" path="/var/lib/kubelet/pods/77e791b9-b21f-47ec-a85a-ec8b626d85ce/volumes" Nov 03 04:08:38.652390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.652319 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:08:38.775832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.775792 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.775908 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.775968 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776067 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776114 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776131 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:38.776184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776176 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776223 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:38.776410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776265 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776292 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776327 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\") pod \"cfc671a7-f4ac-47f6-81e6-79194593da5a\" (UID: \"cfc671a7-f4ac-47f6-81e6-79194593da5a\") " Nov 03 04:08:38.776636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776459 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:38.777489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776891 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.777489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776915 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.777489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.776928 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.777489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.777229 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:38.777489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.777457 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:38.779659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.778942 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:38.779659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.779267 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:08:38.779659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.779282 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:08:38.780531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.780499 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk" pod "cfc671a7-f4ac-47f6-81e6-79194593da5a" (UID: "cfc671a7-f4ac-47f6-81e6-79194593da5a"). InnerVolumeSpecName "kube-api-access-zzqnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:38.878436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878391 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.878436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878436 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.878663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878452 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc671a7-f4ac-47f6-81e6-79194593da5a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.878663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878467 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc671a7-f4ac-47f6-81e6-79194593da5a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.878663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878482 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqnk\" (UniqueName: \"kubernetes.io/projected/cfc671a7-f4ac-47f6-81e6-79194593da5a-kube-api-access-zzqnk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:38.878663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:38.878497 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc671a7-f4ac-47f6-81e6-79194593da5a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:39.192685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.192653 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:08:39.217666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.217636 5275 generic.go:356] "Generic (PLEG): container finished" podID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerID="af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd" exitCode=0 Nov 03 04:08:39.217880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.217705 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" Nov 03 04:08:39.217880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.217722 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerDied","Data":"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd"} Nov 03 04:08:39.217880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.217794 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk" event={"ID":"cfc671a7-f4ac-47f6-81e6-79194593da5a","Type":"ContainerDied","Data":"efb7ed7ed78e2a62500f161fcff19ab2ff0fd9166e01c1b4a2071621f1c14a89"} Nov 03 04:08:39.217880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.217824 5275 scope.go:122] "RemoveContainer" containerID="af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd" Nov 03 04:08:39.228673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.228652 5275 scope.go:122] "RemoveContainer" containerID="62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6" Nov 03 04:08:39.239250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.239230 5275 scope.go:122] "RemoveContainer" containerID="af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd" Nov 03 04:08:39.239589 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:39.239567 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd\": container with ID starting with af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd not found: ID does not exist" containerID="af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd" Nov 03 04:08:39.239634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.239597 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd"} err="failed to get container status \"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd\": rpc error: code = NotFound desc = could not find container \"af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd\": container with ID starting with af1a9dadc9f87e61283780668920505b36ff5f9bc29f3643243b894e166c43bd not found: ID does not exist" Nov 03 04:08:39.239634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.239615 5275 scope.go:122] "RemoveContainer" containerID="62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6" Nov 03 04:08:39.239908 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:08:39.239886 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6\": container with ID starting with 62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6 not found: ID does not exist" containerID="62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6" Nov 03 04:08:39.239962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.239915 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6"} err="failed to get container status \"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6\": rpc error: code = NotFound desc = could not find container \"62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6\": container with ID starting with 62ead57c28e1579f279ec75eeb7e4856ba578f273e4349698e80676b241653a6 not found: ID does not exist" Nov 03 04:08:39.429208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.429169 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:08:39.435362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:39.434195 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c832d-openshift-default-66779dc9b9-l7cfk"] Nov 03 04:08:40.986977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:40.986913 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" path="/var/lib/kubelet/pods/cfc671a7-f4ac-47f6-81e6-79194593da5a/volumes" Nov 03 04:08:43.414198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.414154 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:08:43.415106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415085 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:43.415171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415108 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:43.415171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415135 5275 cpu_manager.go:414] "containerMap: removing container" podUID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" containerName="payload-processing" Nov 03 04:08:43.415171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415140 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" containerName="payload-processing" Nov 03 04:08:43.415171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415150 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" Nov 03 04:08:43.415171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415156 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" Nov 03 04:08:43.415347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415288 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:43.415347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415300 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e791b9-b21f-47ec-a85a-ec8b626d85ce" containerName="payload-processing" Nov 03 04:08:43.415347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.415308 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa17b61-8bab-4fa0-a556-6dd919a79ec2" containerName="istio-proxy" Nov 03 04:08:43.420686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.420661 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.476054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.476007 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:08:43.521090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.521059 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-934bd4\"" Nov 03 04:08:43.521090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.521081 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-934bd4-dockercfg-2dvms\"" Nov 03 04:08:43.557076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.557030 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.557347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.557205 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65dt\" (UniqueName: \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.670875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.670764 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.671071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.670952 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65dt\" (UniqueName: \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.672613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.671830 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.731902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.731867 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65dt\" (UniqueName: \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\") pod \"payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:43.736649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:43.736628 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:44.117287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:44.117261 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:08:44.118655 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:44.118627 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9af281c_5adb_40f0_92e6_6c46cc79fe9e.slice/crio-1e37b84fc0a5f32bea525c6ac7beee099c6ff59a3405a02da518d47d637a07fc WatchSource:0}: Error finding container 1e37b84fc0a5f32bea525c6ac7beee099c6ff59a3405a02da518d47d637a07fc: Status 404 returned error can't find the container with id 1e37b84fc0a5f32bea525c6ac7beee099c6ff59a3405a02da518d47d637a07fc Nov 03 04:08:44.264240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:44.264201 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" event={"ID":"a9af281c-5adb-40f0-92e6-6c46cc79fe9e","Type":"ContainerStarted","Data":"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7"} Nov 03 04:08:44.264457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:44.264267 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" event={"ID":"a9af281c-5adb-40f0-92e6-6c46cc79fe9e","Type":"ContainerStarted","Data":"1e37b84fc0a5f32bea525c6ac7beee099c6ff59a3405a02da518d47d637a07fc"} Nov 03 04:08:44.264457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:44.264399 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:44.382131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:44.382079 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" podStartSLOduration=1.382059833 podStartE2EDuration="1.382059833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:44.379216316 +0000 UTC m=+4015.531904738" watchObservedRunningTime="2026-11-03 04:08:44.382059833 +0000 UTC m=+4015.534748254" Nov 03 04:08:45.965949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:45.965913 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:08:45.966616 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:45.966601 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:45.966616 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:45.966618 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:45.966766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:45.966744 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc671a7-f4ac-47f6-81e6-79194593da5a" containerName="istio-proxy" Nov 03 04:08:45.970485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:45.970466 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.016701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.016665 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtplp\" (UniqueName: \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.017099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.017070 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.080871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.078311 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-eed598\"" Nov 03 04:08:46.080871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.078499 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-eed598-dockercfg-bz2f6\"" Nov 03 04:08:46.102874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.092133 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:08:46.110887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.106925 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:08:46.135870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.124980 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtplp\" (UniqueName: \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.135870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.125118 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.135870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.126263 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.135870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.127047 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.183178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.183142 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:08:46.190612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.190581 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtplp\" (UniqueName: \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\") pod \"payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.226906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.226788 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjn5r\" (UniqueName: \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.227064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.227043 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.286042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.286004 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:46.328006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.327965 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjn5r\" (UniqueName: \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.328244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.328221 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.329003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.328983 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.389466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.389434 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjn5r\" (UniqueName: \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\") pod \"payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.470625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.470589 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:46.671935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.671904 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:08:46.672973 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:46.672947 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432b9f8f_e7a8_42d2_8b05_6810011465de.slice/crio-b813c3bb759bd4aba5f6d2dfcd57598305a1daded97e011a807fc22ceccdb221 WatchSource:0}: Error finding container b813c3bb759bd4aba5f6d2dfcd57598305a1daded97e011a807fc22ceccdb221: Status 404 returned error can't find the container with id b813c3bb759bd4aba5f6d2dfcd57598305a1daded97e011a807fc22ceccdb221 Nov 03 04:08:46.854309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:46.854279 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:08:46.855171 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:08:46.855149 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22f6d21_5909_4f26_b141_5d8e796d9714.slice/crio-4375273e455b615d7f9455f9679da16eaa3ce2212b306e37f448ac2062749081 WatchSource:0}: Error finding container 4375273e455b615d7f9455f9679da16eaa3ce2212b306e37f448ac2062749081: Status 404 returned error can't find the container with id 4375273e455b615d7f9455f9679da16eaa3ce2212b306e37f448ac2062749081 Nov 03 04:08:47.290487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.290449 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" event={"ID":"432b9f8f-e7a8-42d2-8b05-6810011465de","Type":"ContainerStarted","Data":"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd"} Nov 03 04:08:47.291014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.290516 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" event={"ID":"432b9f8f-e7a8-42d2-8b05-6810011465de","Type":"ContainerStarted","Data":"b813c3bb759bd4aba5f6d2dfcd57598305a1daded97e011a807fc22ceccdb221"} Nov 03 04:08:47.291014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.290549 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:08:47.292189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.292165 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" event={"ID":"c22f6d21-5909-4f26-b141-5d8e796d9714","Type":"ContainerStarted","Data":"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42"} Nov 03 04:08:47.292306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.292207 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" event={"ID":"c22f6d21-5909-4f26-b141-5d8e796d9714","Type":"ContainerStarted","Data":"4375273e455b615d7f9455f9679da16eaa3ce2212b306e37f448ac2062749081"} Nov 03 04:08:47.292306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.292265 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:47.408702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:47.408645 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" podStartSLOduration=2.408631221 podStartE2EDuration="2.408631221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:47.405503195 +0000 UTC m=+4018.558191616" watchObservedRunningTime="2026-11-03 04:08:47.408631221 +0000 UTC m=+4018.561319642" Nov 03 04:08:53.598600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.598568 5275 scope.go:122] "RemoveContainer" containerID="ef92495e503ac9182ea8d94193dcaaa9731249d268e5009a2f379d304bd75595" Nov 03 04:08:53.609521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.609501 5275 scope.go:122] "RemoveContainer" containerID="6f3467d0a30e44f267112d797244360ba47ad261d5dd1b10c1ec89187fc19707" Nov 03 04:08:53.619349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.619323 5275 scope.go:122] "RemoveContainer" containerID="3b1a8aaf596f381dc69f9b5ab055657b717af1e639c8a1a8d5be845a4e254d4e" Nov 03 04:08:53.629381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.629360 5275 scope.go:122] "RemoveContainer" containerID="40f4af9d963a646722b81c57407399b13590979d855e2609c5b12471fcacb9b7" Nov 03 04:08:53.639345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.639319 5275 scope.go:122] "RemoveContainer" containerID="7f3ec21663094561bffb2c1f5292831ad163ac1d4b13b4159e96234661d3c6d4" Nov 03 04:08:53.649149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.649127 5275 scope.go:122] "RemoveContainer" containerID="acbfc0703f6166ee6aaa9be15bbf79a6d05186d06667ce8652ac5fe326614481" Nov 03 04:08:53.658889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.658833 5275 scope.go:122] "RemoveContainer" containerID="c25d9e0b3c59407602242002b940d4f90187875b64c4dd42867363c185b7c8d6" Nov 03 04:08:53.668769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:53.668748 5275 scope.go:122] "RemoveContainer" containerID="eb6b23a22902c5a0e8b4e2e2e4c22ab9d40fdace0b6789b0c4f59b20bd6fba44" Nov 03 04:08:55.275914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:55.275881 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:08:55.392867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:55.392790 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" podStartSLOduration=9.392774319 podStartE2EDuration="9.392774319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:47.526861688 +0000 UTC m=+4018.679550106" watchObservedRunningTime="2026-11-03 04:08:55.392774319 +0000 UTC m=+4026.545462796" Nov 03 04:08:58.303384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:58.303351 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:08:58.303934 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:08:58.303438 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:09:02.331542 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:09:02.331500 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:09:32.330470 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:09:32.330375 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:09:55.831071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:55.831031 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:09:55.831634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:55.831101 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:09:58.917251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.917219 5275 generic.go:356] "Generic (PLEG): container finished" podID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerID="f5a3da2b1b42a04526626524c0151ff0b60fd694610ac9bb6fedb3c07268203f" exitCode=0 Nov 03 04:09:58.917716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.917295 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerDied","Data":"f5a3da2b1b42a04526626524c0151ff0b60fd694610ac9bb6fedb3c07268203f"} Nov 03 04:09:58.917775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.917727 5275 scope.go:122] "RemoveContainer" containerID="f5a3da2b1b42a04526626524c0151ff0b60fd694610ac9bb6fedb3c07268203f" Nov 03 04:09:58.919472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.919439 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:09:58.919596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.919514 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:09:58.919596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:58.919551 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:09:59.925813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.925729 5275 generic.go:356] "Generic (PLEG): container finished" podID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerID="4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97" exitCode=0 Nov 03 04:09:59.926394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.925812 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerDied","Data":"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97"} Nov 03 04:09:59.926394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.926370 5275 scope.go:122] "RemoveContainer" containerID="4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97" Nov 03 04:09:59.928185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.928162 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerStarted","Data":"70dd152f18d8504996a52031a5f3be42197039d4af416177ed78bf9c39ebf951"} Nov 03 04:09:59.928299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.928277 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:09:59.928363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.928342 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:09:59.928410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:09:59.928374 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:10:00.383562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:00.383477 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:10:00.383787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:00.383693 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 04:10:00.383787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:00.383767 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 04:10:00.936799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:00.936750 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerStarted","Data":"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7"} Nov 03 04:10:01.411017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:01.410317 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 04:10:01.411017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:01.410379 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 04:10:01.775872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:01.774245 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:10:01.775872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:01.774380 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 04:10:01.775872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:01.774423 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 04:10:02.332294 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:10:02.332259 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:10:02.385871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:02.385256 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 04:10:02.385871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:02.385324 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 04:10:02.792872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:02.775300 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 04:10:02.792872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:02.775359 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 04:10:03.385874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:03.385093 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 04:10:03.385874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:03.385156 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 04:10:03.782876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:03.782811 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 04:10:03.783161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:03.783130 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 04:10:04.387040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:04.387004 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 04:10:04.387664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:04.387634 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 04:10:04.774454 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:04.774412 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 04:10:04.774765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:04.774735 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 04:10:05.501476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:05.501439 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:10:05.503121 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:05.503086 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:10:05.504127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:05.504097 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:10:05.774258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:05.774222 5275 patch_prober.go:29] interesting pod/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 04:10:05.774456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:05.774275 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 04:10:06.779390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:06.779358 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:10:06.779978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:06.779880 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:10:06.780839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:06.780813 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:10:28.822337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:28.822295 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:10:28.822787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:28.822375 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:10:29.247814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.247723 5275 generic.go:356] "Generic (PLEG): container finished" podID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerID="83d453970edaa910f9f1a96550e50acbb2db0baa2586e4e61e8c340e42321963" exitCode=0 Nov 03 04:10:29.247814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.247788 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerDied","Data":"83d453970edaa910f9f1a96550e50acbb2db0baa2586e4e61e8c340e42321963"} Nov 03 04:10:29.248318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.248302 5275 scope.go:122] "RemoveContainer" containerID="83d453970edaa910f9f1a96550e50acbb2db0baa2586e4e61e8c340e42321963" Nov 03 04:10:29.250548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.250510 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:10:29.250672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.250605 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:10:29.250672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:29.250653 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:10:30.282872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:30.282181 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerStarted","Data":"dfd9492b2667da5529c5e42fdc82c06900dcf569a3b0c9cc4b6c6130afb3e6d7"} Nov 03 04:10:31.142534 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:31.142495 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:10:31.142879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:31.142830 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 03 04:10:31.142935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:31.142892 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 03 04:10:32.142450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:32.142412 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 03 04:10:32.142907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:32.142482 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 03 04:10:32.337956 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:10:32.337910 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:10:33.142293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:33.142255 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 03 04:10:33.142489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:33.142313 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 03 04:10:34.142423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:34.142384 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 03 04:10:34.142625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:34.142442 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 03 04:10:35.142821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:35.142783 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 03 04:10:35.143269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:35.142871 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 03 04:10:36.147624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:36.147589 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:10:36.148128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:36.148106 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:10:36.148822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:10:36.148800 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:11:02.334289 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:11:02.334245 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:11:03.237306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.237262 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:11:03.241110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.241087 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.383750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.383702 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386469 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386541 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386600 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386630 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386661 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386709 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.386831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386797 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.390237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.386903 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpz2\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.405364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.405328 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8540f6-openshift-default-dockercfg-pv8rg\"" Nov 03 04:11:03.423517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.423418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:11:03.488003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.487898 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpz2\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488010 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488101 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488141 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488190 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488219 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488253 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488296 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488368 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.488816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.488789 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.489078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.489057 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.489729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.489701 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.490056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.490031 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.490155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.490127 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.492360 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.492329 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.500673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.500644 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.551731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.551698 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.551941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.551921 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpz2\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:03.563917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:03.563868 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:04.010604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.010570 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:11:04.011726 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:11:04.011696 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d45890_5837_4fd6_a52b_abce2f3b59c3.slice/crio-96deb5f12b4a0b24cb0c00d3c4de6bb72e70cca80a4fb63f78d4cd094c0a523b WatchSource:0}: Error finding container 96deb5f12b4a0b24cb0c00d3c4de6bb72e70cca80a4fb63f78d4cd094c0a523b: Status 404 returned error can't find the container with id 96deb5f12b4a0b24cb0c00d3c4de6bb72e70cca80a4fb63f78d4cd094c0a523b Nov 03 04:11:04.014488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.014457 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:11:04.014565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.014524 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:11:04.014613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.014553 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:11:04.548453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.548414 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerStarted","Data":"a18639815908874aa320ef5ae8843367e37fc745547e534af3cfb012d342d6a6"} Nov 03 04:11:04.548880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.548515 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerStarted","Data":"96deb5f12b4a0b24cb0c00d3c4de6bb72e70cca80a4fb63f78d4cd094c0a523b"} Nov 03 04:11:04.564789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.564754 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:04.679521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:04.678831 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podStartSLOduration=1.678807615 podStartE2EDuration="1.678807615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:11:04.668200569 +0000 UTC m=+4155.820888990" watchObservedRunningTime="2026-11-03 04:11:04.678807615 +0000 UTC m=+4155.831496037" Nov 03 04:11:05.570996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:05.570966 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:06.564911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:06.564879 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:06.565936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:06.565913 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:11:11.382885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.382825 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:11:11.386504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.386479 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.439263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.439228 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8540f6\"" Nov 03 04:11:11.444701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.444664 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:11:11.562548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.562514 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqrt\" (UniqueName: \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.563035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.563009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.566750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.566722 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:11:11.572199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.572171 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.627474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.627372 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8540f6-dockercfg-wmzr7\"" Nov 03 04:11:11.647717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.647640 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:11:11.664783 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.664743 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqrt\" (UniqueName: \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.665044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.665023 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.666076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.666050 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.726055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.726021 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqrt\" (UniqueName: \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\") pod \"payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:11.766364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.766324 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.766593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.766393 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sb4\" (UniqueName: \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.868529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.868478 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.868764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.868586 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sb4\" (UniqueName: \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.869307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.869280 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.929859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.929773 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sb4\" (UniqueName: \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\") pod \"payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:11.963660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:11.963620 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:12.003533 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.003492 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:12.355452 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:11:12.355419 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fb8aa4_6e7a_4570_904b_579022f834d7.slice/crio-0ac685e9eaf0ff113c9acff0426ed2f65f4db5cf14d38a3fbc439834bb1b62ed WatchSource:0}: Error finding container 0ac685e9eaf0ff113c9acff0426ed2f65f4db5cf14d38a3fbc439834bb1b62ed: Status 404 returned error can't find the container with id 0ac685e9eaf0ff113c9acff0426ed2f65f4db5cf14d38a3fbc439834bb1b62ed Nov 03 04:11:12.355816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.355789 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:11:12.466199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.456574 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:11:12.646903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.646789 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" event={"ID":"cfd810c6-10f1-4b67-837f-a7d3e0858e6c","Type":"ContainerStarted","Data":"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a"} Nov 03 04:11:12.646903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.646877 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" event={"ID":"cfd810c6-10f1-4b67-837f-a7d3e0858e6c","Type":"ContainerStarted","Data":"8f1839a1aa35f69ee22d7b3ca476d7d73f4b3337909d2747b5008d4c0f6e15c3"} Nov 03 04:11:12.647152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.647045 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:12.648861 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.648820 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" event={"ID":"38fb8aa4-6e7a-4570-904b-579022f834d7","Type":"ContainerStarted","Data":"f043e1704daa3804cccd5a8eb9e04936c7b759ced25844e4217933c26ccfddbb"} Nov 03 04:11:12.649030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.648891 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:12.649030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.648911 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" event={"ID":"38fb8aa4-6e7a-4570-904b-579022f834d7","Type":"ContainerStarted","Data":"0ac685e9eaf0ff113c9acff0426ed2f65f4db5cf14d38a3fbc439834bb1b62ed"} Nov 03 04:11:12.889770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:12.888109 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" podStartSLOduration=1.888085535 podStartE2EDuration="1.888085535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:11:12.76439102 +0000 UTC m=+4163.917079443" watchObservedRunningTime="2026-11-03 04:11:12.888085535 +0000 UTC m=+4164.040773969" Nov 03 04:11:16.371063 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:16.371009 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" podStartSLOduration=5.370990586 podStartE2EDuration="5.370990586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:11:12.892647419 +0000 UTC m=+4164.045335842" watchObservedRunningTime="2026-11-03 04:11:16.370990586 +0000 UTC m=+4167.523679008" Nov 03 04:11:16.374970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:16.374942 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:11:16.486726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:16.486682 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:11:21.381880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.376925 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" podUID="2c358ccf-6162-45f8-aad6-3113d9b673d3" containerName="payload-processing" containerID="cri-o://24542c35e00d34c7c4d496d5e4394ab17a5323194c1f09e03d5411f630ec3d2d" gracePeriod=25 Nov 03 04:11:21.487511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.487452 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" podUID="d5807b36-d606-45ba-a992-3bfff331e5d3" containerName="payload-pre-processing" containerID="cri-o://a7ab99f18c1c23c2f6acf7f5a82bc7c7f4c32855db813f49a4e8e291e82af980" gracePeriod=25 Nov 03 04:11:21.728641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.728605 5275 generic.go:356] "Generic (PLEG): container finished" podID="2c358ccf-6162-45f8-aad6-3113d9b673d3" containerID="24542c35e00d34c7c4d496d5e4394ab17a5323194c1f09e03d5411f630ec3d2d" exitCode=0 Nov 03 04:11:21.729088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.729057 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" event={"ID":"2c358ccf-6162-45f8-aad6-3113d9b673d3","Type":"ContainerDied","Data":"24542c35e00d34c7c4d496d5e4394ab17a5323194c1f09e03d5411f630ec3d2d"} Nov 03 04:11:21.731074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.731051 5275 generic.go:356] "Generic (PLEG): container finished" podID="d5807b36-d606-45ba-a992-3bfff331e5d3" containerID="a7ab99f18c1c23c2f6acf7f5a82bc7c7f4c32855db813f49a4e8e291e82af980" exitCode=0 Nov 03 04:11:21.731204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.731126 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" event={"ID":"d5807b36-d606-45ba-a992-3bfff331e5d3","Type":"ContainerDied","Data":"a7ab99f18c1c23c2f6acf7f5a82bc7c7f4c32855db813f49a4e8e291e82af980"} Nov 03 04:11:21.763309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.763280 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:11:21.852892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.852820 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\" (UniqueName: \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\") pod \"2c358ccf-6162-45f8-aad6-3113d9b673d3\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " Nov 03 04:11:21.853086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.853057 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\") pod \"2c358ccf-6162-45f8-aad6-3113d9b673d3\" (UID: \"2c358ccf-6162-45f8-aad6-3113d9b673d3\") " Nov 03 04:11:21.853473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.853442 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume" pod "2c358ccf-6162-45f8-aad6-3113d9b673d3" (UID: "2c358ccf-6162-45f8-aad6-3113d9b673d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:21.854402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.854380 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:11:21.855121 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.855100 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l" pod "2c358ccf-6162-45f8-aad6-3113d9b673d3" (UID: "2c358ccf-6162-45f8-aad6-3113d9b673d3"). InnerVolumeSpecName "kube-api-access-ptz8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:21.954344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954306 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\") pod \"d5807b36-d606-45ba-a992-3bfff331e5d3\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " Nov 03 04:11:21.954559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954430 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\" (UniqueName: \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\") pod \"d5807b36-d606-45ba-a992-3bfff331e5d3\" (UID: \"d5807b36-d606-45ba-a992-3bfff331e5d3\") " Nov 03 04:11:21.954689 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954665 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume" pod "d5807b36-d606-45ba-a992-3bfff331e5d3" (UID: "d5807b36-d606-45ba-a992-3bfff331e5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:21.954962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954948 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c358ccf-6162-45f8-aad6-3113d9b673d3-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:21.955015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954965 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5807b36-d606-45ba-a992-3bfff331e5d3-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:21.955015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.954975 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptz8l\" (UniqueName: \"kubernetes.io/projected/2c358ccf-6162-45f8-aad6-3113d9b673d3-kube-api-access-ptz8l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:21.956599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:21.956578 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh" pod "d5807b36-d606-45ba-a992-3bfff331e5d3" (UID: "d5807b36-d606-45ba-a992-3bfff331e5d3"). InnerVolumeSpecName "kube-api-access-pmchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:22.056927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.056886 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmchh\" (UniqueName: \"kubernetes.io/projected/d5807b36-d606-45ba-a992-3bfff331e5d3-kube-api-access-pmchh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:22.740736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.740652 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" Nov 03 04:11:22.740736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.740683 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529" event={"ID":"2c358ccf-6162-45f8-aad6-3113d9b673d3","Type":"ContainerDied","Data":"35d5c1b66036fcfe64eef9170b63f6795dfcd154067a9b1abc8aa672a1e1e4b8"} Nov 03 04:11:22.741317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.740745 5275 scope.go:122] "RemoveContainer" containerID="24542c35e00d34c7c4d496d5e4394ab17a5323194c1f09e03d5411f630ec3d2d" Nov 03 04:11:22.742430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.742400 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" event={"ID":"d5807b36-d606-45ba-a992-3bfff331e5d3","Type":"ContainerDied","Data":"eb058e38b599b9b80d9821651f771829492ab117437585d018385a13d5aac7e1"} Nov 03 04:11:22.742559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.742406 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m" Nov 03 04:11:22.754797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.754772 5275 scope.go:122] "RemoveContainer" containerID="a7ab99f18c1c23c2f6acf7f5a82bc7c7f4c32855db813f49a4e8e291e82af980" Nov 03 04:11:22.911348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.911314 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:11:22.914901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.914873 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-214eab-64856d5cbc-sj529"] Nov 03 04:11:22.986411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:22.986372 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c358ccf-6162-45f8-aad6-3113d9b673d3" path="/var/lib/kubelet/pods/2c358ccf-6162-45f8-aad6-3113d9b673d3/volumes" Nov 03 04:11:23.028080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:23.028050 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:11:23.031368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:23.031343 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-214eab-86577df6cf-t996m"] Nov 03 04:11:23.661456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:23.661426 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:11:23.661949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:23.661925 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:11:24.986814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:24.986781 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5807b36-d606-45ba-a992-3bfff331e5d3" path="/var/lib/kubelet/pods/d5807b36-d606-45ba-a992-3bfff331e5d3/volumes" Nov 03 04:11:32.334658 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:11:32.334606 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:11:42.177035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:42.176992 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:11:45.229775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:45.229741 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:11:45.335695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:45.335660 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:11:47.178130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.178081 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" podUID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" containerName="payload-pre-processing" containerID="cri-o://f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7" gracePeriod=25 Nov 03 04:11:47.479290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.479259 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:11:47.526512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.526475 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\" (UniqueName: \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\") pod \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " Nov 03 04:11:47.526722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.526581 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\") pod \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\" (UID: \"a9af281c-5adb-40f0-92e6-6c46cc79fe9e\") " Nov 03 04:11:47.527023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.527001 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume" pod "a9af281c-5adb-40f0-92e6-6c46cc79fe9e" (UID: "a9af281c-5adb-40f0-92e6-6c46cc79fe9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:47.528796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.528771 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt" pod "a9af281c-5adb-40f0-92e6-6c46cc79fe9e" (UID: "a9af281c-5adb-40f0-92e6-6c46cc79fe9e"). InnerVolumeSpecName "kube-api-access-h65dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:47.628609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.628563 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h65dt\" (UniqueName: \"kubernetes.io/projected/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-kube-api-access-h65dt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:47.628609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.628608 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9af281c-5adb-40f0-92e6-6c46cc79fe9e-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:47.956054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.956015 5275 generic.go:356] "Generic (PLEG): container finished" podID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" containerID="f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7" exitCode=0 Nov 03 04:11:47.956266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.956063 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" Nov 03 04:11:47.956266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.956097 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" event={"ID":"a9af281c-5adb-40f0-92e6-6c46cc79fe9e","Type":"ContainerDied","Data":"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7"} Nov 03 04:11:47.956266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.956168 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg" event={"ID":"a9af281c-5adb-40f0-92e6-6c46cc79fe9e","Type":"ContainerDied","Data":"1e37b84fc0a5f32bea525c6ac7beee099c6ff59a3405a02da518d47d637a07fc"} Nov 03 04:11:47.956266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.956196 5275 scope.go:122] "RemoveContainer" containerID="f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7" Nov 03 04:11:47.967681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.967662 5275 scope.go:122] "RemoveContainer" containerID="f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7" Nov 03 04:11:47.968057 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:11:47.968033 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7\": container with ID starting with f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7 not found: ID does not exist" containerID="f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7" Nov 03 04:11:47.968133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:47.968064 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7"} err="failed to get container status \"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7\": rpc error: code = NotFound desc = could not find container \"f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7\": container with ID starting with f04046ef8fd2e4e2dfc47a6f0f0fd7d8dbc8153c11c3c15e9bf54a7e7eec68a7 not found: ID does not exist" Nov 03 04:11:48.122036 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:48.122000 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:11:48.173737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:48.173698 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-934bd4-7c8ff85cf5-2srpg"] Nov 03 04:11:48.985996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:48.985962 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" path="/var/lib/kubelet/pods/a9af281c-5adb-40f0-92e6-6c46cc79fe9e/volumes" Nov 03 04:11:50.233743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.233659 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" podUID="c22f6d21-5909-4f26-b141-5d8e796d9714" containerName="payload-processing" containerID="cri-o://e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42" gracePeriod=25 Nov 03 04:11:50.304258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.304234 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:11:50.330964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.330939 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:11:50.336422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.336391 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" containerName="payload-pre-processing" containerID="cri-o://799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd" gracePeriod=25 Nov 03 04:11:50.520635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.520601 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:11:50.568136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.568103 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\" (UniqueName: \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\") pod \"c22f6d21-5909-4f26-b141-5d8e796d9714\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " Nov 03 04:11:50.568356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.568158 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\") pod \"c22f6d21-5909-4f26-b141-5d8e796d9714\" (UID: \"c22f6d21-5909-4f26-b141-5d8e796d9714\") " Nov 03 04:11:50.568661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.568630 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume" pod "c22f6d21-5909-4f26-b141-5d8e796d9714" (UID: "c22f6d21-5909-4f26-b141-5d8e796d9714"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:50.568799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.568776 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22f6d21-5909-4f26-b141-5d8e796d9714-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:50.570991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.570960 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r" pod "c22f6d21-5909-4f26-b141-5d8e796d9714" (UID: "c22f6d21-5909-4f26-b141-5d8e796d9714"). InnerVolumeSpecName "kube-api-access-pjn5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:50.624769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.624741 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:11:50.669630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.669593 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\" (UniqueName: \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\") pod \"432b9f8f-e7a8-42d2-8b05-6810011465de\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " Nov 03 04:11:50.669811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.669662 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\") pod \"432b9f8f-e7a8-42d2-8b05-6810011465de\" (UID: \"432b9f8f-e7a8-42d2-8b05-6810011465de\") " Nov 03 04:11:50.670043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.670018 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume" pod "432b9f8f-e7a8-42d2-8b05-6810011465de" (UID: "432b9f8f-e7a8-42d2-8b05-6810011465de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:50.670258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.670238 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432b9f8f-e7a8-42d2-8b05-6810011465de-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:50.670370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.670264 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjn5r\" (UniqueName: \"kubernetes.io/projected/c22f6d21-5909-4f26-b141-5d8e796d9714-kube-api-access-pjn5r\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:50.672120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.672095 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp" pod "432b9f8f-e7a8-42d2-8b05-6810011465de" (UID: "432b9f8f-e7a8-42d2-8b05-6810011465de"). InnerVolumeSpecName "kube-api-access-jtplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:50.771636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.771606 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtplp\" (UniqueName: \"kubernetes.io/projected/432b9f8f-e7a8-42d2-8b05-6810011465de-kube-api-access-jtplp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:50.980751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.980718 5275 generic.go:356] "Generic (PLEG): container finished" podID="c22f6d21-5909-4f26-b141-5d8e796d9714" containerID="e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42" exitCode=0 Nov 03 04:11:50.980976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.980834 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" Nov 03 04:11:50.982499 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.982471 5275 generic.go:356] "Generic (PLEG): container finished" podID="432b9f8f-e7a8-42d2-8b05-6810011465de" containerID="799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd" exitCode=0 Nov 03 04:11:50.982667 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.982530 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:11:50.986799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.986764 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" event={"ID":"c22f6d21-5909-4f26-b141-5d8e796d9714","Type":"ContainerDied","Data":"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42"} Nov 03 04:11:50.986951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.986828 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m" event={"ID":"c22f6d21-5909-4f26-b141-5d8e796d9714","Type":"ContainerDied","Data":"4375273e455b615d7f9455f9679da16eaa3ce2212b306e37f448ac2062749081"} Nov 03 04:11:50.986951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.986874 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" event={"ID":"432b9f8f-e7a8-42d2-8b05-6810011465de","Type":"ContainerDied","Data":"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd"} Nov 03 04:11:50.986951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.986873 5275 scope.go:122] "RemoveContainer" containerID="e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42" Nov 03 04:11:50.986951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:50.986901 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" event={"ID":"432b9f8f-e7a8-42d2-8b05-6810011465de","Type":"ContainerDied","Data":"b813c3bb759bd4aba5f6d2dfcd57598305a1daded97e011a807fc22ceccdb221"} Nov 03 04:11:51.000312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.000288 5275 scope.go:122] "RemoveContainer" containerID="e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42" Nov 03 04:11:51.000646 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:11:51.000626 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42\": container with ID starting with e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42 not found: ID does not exist" containerID="e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42" Nov 03 04:11:51.000721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.000655 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42"} err="failed to get container status \"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42\": rpc error: code = NotFound desc = could not find container \"e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42\": container with ID starting with e6db74236f3890d8bf84f221c392e3a461f42fdc4ec8f0566ed048ce4e3b9f42 not found: ID does not exist" Nov 03 04:11:51.000721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.000673 5275 scope.go:122] "RemoveContainer" containerID="799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd" Nov 03 04:11:51.010586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.010565 5275 scope.go:122] "RemoveContainer" containerID="799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd" Nov 03 04:11:51.010903 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:11:51.010884 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd\": container with ID starting with 799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd not found: ID does not exist" containerID="799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd" Nov 03 04:11:51.010967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.010910 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd"} err="failed to get container status \"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd\": rpc error: code = NotFound desc = could not find container \"799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd\": container with ID starting with 799de62cec8d4dd1b59ab273779a68e6da53bec91605fc2fe89ccfd9e4dfadfd not found: ID does not exist" Nov 03 04:11:51.199475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.199389 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:11:51.202550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:51.202529 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eed598-6f487679d5-qz64m"] Nov 03 04:11:52.986402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:11:52.986369 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22f6d21-5909-4f26-b141-5d8e796d9714" path="/var/lib/kubelet/pods/c22f6d21-5909-4f26-b141-5d8e796d9714/volumes" Nov 03 04:12:02.331839 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:12:02.331802 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:12:20.994442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:20.994387 5275 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod432b9f8f-e7a8-42d2-8b05-6810011465de"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod432b9f8f-e7a8-42d2-8b05-6810011465de] : Timed out while waiting for systemd to remove kubepods-burstable-pod432b9f8f_e7a8_42d2_8b05_6810011465de.slice" Nov 03 04:12:20.994442 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:12:20.994445 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod432b9f8f-e7a8-42d2-8b05-6810011465de] : unable to destroy cgroup paths for cgroup [kubepods burstable pod432b9f8f-e7a8-42d2-8b05-6810011465de] : Timed out while waiting for systemd to remove kubepods-burstable-pod432b9f8f_e7a8_42d2_8b05_6810011465de.slice" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" Nov 03 04:12:21.205249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:21.205214 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk" Nov 03 04:12:21.436006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:21.435894 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:12:21.440286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:21.440248 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eed598-c556fc7c5-g8pdk"] Nov 03 04:12:22.985735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:22.985658 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" path="/var/lib/kubelet/pods/432b9f8f-e7a8-42d2-8b05-6810011465de/volumes" Nov 03 04:12:32.337918 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:12:32.337841 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:12:42.858499 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:42.858461 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:12:42.858950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:42.858770 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" containerID="cri-o://6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7" gracePeriod=30 Nov 03 04:12:47.232505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:47.232467 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:12:47.232963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:47.232691 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" containerID="cri-o://dfd9492b2667da5529c5e42fdc82c06900dcf569a3b0c9cc4b6c6130afb3e6d7" gracePeriod=30 Nov 03 04:12:48.158091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.158062 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:12:48.246633 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246598 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246683 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246708 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246770 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246840 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246910 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.246963 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247023 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.247537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247153 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:12:48.247537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247363 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:48.247654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247531 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:48.247654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247595 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.247654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247618 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab67cc29-e51f-483e-9fde-3e60da64574b-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.247841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.247717 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:48.249736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.249710 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:48.258317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.258283 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:48.258557 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.258523 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "kube-api-access-gkl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:48.259091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.258939 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:12:48.348458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348420 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\") pod \"ab67cc29-e51f-483e-9fde-3e60da64574b\" (UID: \"ab67cc29-e51f-483e-9fde-3e60da64574b\") " Nov 03 04:12:48.348688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348667 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket" pod "ab67cc29-e51f-483e-9fde-3e60da64574b" (UID: "ab67cc29-e51f-483e-9fde-3e60da64574b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:48.348978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348963 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348980 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348990 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.348998 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.349008 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkl29\" (UniqueName: \"kubernetes.io/projected/ab67cc29-e51f-483e-9fde-3e60da64574b-kube-api-access-gkl29\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.349016 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.349049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.349024 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab67cc29-e51f-483e-9fde-3e60da64574b-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:48.411959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.411866 5275 generic.go:356] "Generic (PLEG): container finished" podID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerID="6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7" exitCode=0 Nov 03 04:12:48.411959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.411931 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" Nov 03 04:12:48.411959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.411946 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerDied","Data":"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7"} Nov 03 04:12:48.412204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.412016 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6" event={"ID":"ab67cc29-e51f-483e-9fde-3e60da64574b","Type":"ContainerDied","Data":"183c2be180135f03b2ecece4b7535a466796423d00c7ea369ef050feecace17a"} Nov 03 04:12:48.412204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.412048 5275 scope.go:122] "RemoveContainer" containerID="6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7" Nov 03 04:12:48.424400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.424369 5275 scope.go:122] "RemoveContainer" containerID="4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97" Nov 03 04:12:48.434270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.434250 5275 scope.go:122] "RemoveContainer" containerID="6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7" Nov 03 04:12:48.434570 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:12:48.434550 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7\": container with ID starting with 6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7 not found: ID does not exist" containerID="6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7" Nov 03 04:12:48.434634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.434578 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7"} err="failed to get container status \"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7\": rpc error: code = NotFound desc = could not find container \"6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7\": container with ID starting with 6643f71e823a79525551e4c3c9fc456ee01ddcbf33b25a2ad3c854277aa1f6c7 not found: ID does not exist" Nov 03 04:12:48.434634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.434596 5275 scope.go:122] "RemoveContainer" containerID="4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97" Nov 03 04:12:48.434869 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:12:48.434835 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97\": container with ID starting with 4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97 not found: ID does not exist" containerID="4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97" Nov 03 04:12:48.434940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.434879 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97"} err="failed to get container status \"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97\": rpc error: code = NotFound desc = could not find container \"4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97\": container with ID starting with 4e17fbf01e63fcc00ca25e5358a29cfa319fb8f3db9ce85fbeab56c59e9bec97 not found: ID does not exist" Nov 03 04:12:48.582029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.581993 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:12:48.588107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.588077 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-214eab-openshift-default-59849698df-zqrs6"] Nov 03 04:12:48.986331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:48.986301 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" path="/var/lib/kubelet/pods/ab67cc29-e51f-483e-9fde-3e60da64574b/volumes" Nov 03 04:12:50.066058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.066021 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q"] Nov 03 04:12:50.115075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.115039 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4b913-nz59q"] Nov 03 04:12:50.319670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.319589 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:12:50.320535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320518 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c22f6d21-5909-4f26-b141-5d8e796d9714" containerName="payload-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320538 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22f6d21-5909-4f26-b141-5d8e796d9714" containerName="payload-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320561 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d5807b36-d606-45ba-a992-3bfff331e5d3" containerName="payload-pre-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320569 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5807b36-d606-45ba-a992-3bfff331e5d3" containerName="payload-pre-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320580 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2c358ccf-6162-45f8-aad6-3113d9b673d3" containerName="payload-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320589 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c358ccf-6162-45f8-aad6-3113d9b673d3" containerName="payload-processing" Nov 03 04:12:50.320599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320598 5275 cpu_manager.go:414] "containerMap: removing container" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" containerName="payload-pre-processing" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320605 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" containerName="payload-pre-processing" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320628 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320636 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320652 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" containerName="payload-pre-processing" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320660 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" containerName="payload-pre-processing" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320690 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.320805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320699 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320837 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22f6d21-5909-4f26-b141-5d8e796d9714" containerName="payload-processing" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320867 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5807b36-d606-45ba-a992-3bfff331e5d3" containerName="payload-pre-processing" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320875 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c358ccf-6162-45f8-aad6-3113d9b673d3" containerName="payload-processing" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320884 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="432b9f8f-e7a8-42d2-8b05-6810011465de" containerName="payload-pre-processing" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320897 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320906 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9af281c-5adb-40f0-92e6-6c46cc79fe9e" containerName="payload-pre-processing" Nov 03 04:12:50.321033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.320919 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab67cc29-e51f-483e-9fde-3e60da64574b" containerName="istio-proxy" Nov 03 04:12:50.324886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.324833 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380256 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380350 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380383 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380419 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvml\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380477 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380532 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380572 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380594 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.380639 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.387301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.384167 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-255c49-openshift-default-dockercfg-dvtfl\"" Nov 03 04:12:50.424873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.424825 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:12:50.482530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482330 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.482530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482385 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.482530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482436 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvml\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.482530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482487 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.482530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482537 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.483169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482573 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.483169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482596 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.483169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482637 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.483169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.482736 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.483590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.483545 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.485682 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.485657 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.485864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.485762 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.486436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.486407 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.486715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.486694 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.486795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.486718 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.488022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.488001 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.548096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.548067 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvml\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.548427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.548408 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\") pod \"e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.705086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.704987 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:50.986093 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:50.986015 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf89ee6-a59e-48e6-a186-c2bfaf6c011f" path="/var/lib/kubelet/pods/cbf89ee6-a59e-48e6-a186-c2bfaf6c011f/volumes" Nov 03 04:12:51.144086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.144051 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:12:51.150568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.150528 5275 patch_prober.go:29] interesting pod/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:12:51.150722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.150588 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:12:51.179529 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:12:51.179418 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e0c5b8_c155_465a_bf40_16832d2a3ff2.slice/crio-bc7cffd7c17a0229a0f59ad7d6bfdbe4134904cedbf2b7add3f8adae4a18ac9e WatchSource:0}: Error finding container bc7cffd7c17a0229a0f59ad7d6bfdbe4134904cedbf2b7add3f8adae4a18ac9e: Status 404 returned error can't find the container with id bc7cffd7c17a0229a0f59ad7d6bfdbe4134904cedbf2b7add3f8adae4a18ac9e Nov 03 04:12:51.182168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.182133 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:12:51.182290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.182226 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:12:51.182290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.182272 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:12:51.498728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.498621 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerStarted","Data":"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6"} Nov 03 04:12:51.498728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.498688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerStarted","Data":"bc7cffd7c17a0229a0f59ad7d6bfdbe4134904cedbf2b7add3f8adae4a18ac9e"} Nov 03 04:12:51.627037 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.626974 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podStartSLOduration=1.626954349 podStartE2EDuration="1.626954349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:12:51.626547393 +0000 UTC m=+4262.779235814" watchObservedRunningTime="2026-11-03 04:12:51.626954349 +0000 UTC m=+4262.779642778" Nov 03 04:12:51.705945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:51.705908 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:52.509927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.509894 5275 generic.go:356] "Generic (PLEG): container finished" podID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerID="dfd9492b2667da5529c5e42fdc82c06900dcf569a3b0c9cc4b6c6130afb3e6d7" exitCode=0 Nov 03 04:12:52.510364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.509957 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerDied","Data":"dfd9492b2667da5529c5e42fdc82c06900dcf569a3b0c9cc4b6c6130afb3e6d7"} Nov 03 04:12:52.510364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.510049 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" event={"ID":"84c345e5-b98b-4631-ac99-bafe79cccb05","Type":"ContainerDied","Data":"ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb"} Nov 03 04:12:52.510364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.510070 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed2e55d9b1b99c50ca38a350c4b6cae061df694f2b8891d9d9bdc430a50c7edb" Nov 03 04:12:52.510364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.510086 5275 scope.go:122] "RemoveContainer" containerID="83d453970edaa910f9f1a96550e50acbb2db0baa2586e4e61e8c340e42321963" Nov 03 04:12:52.524088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.524069 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:12:52.625984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.625946 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626042 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626137 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626182 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626216 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626300 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626349 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626461 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626442 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:52.626609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626503 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\") pod \"84c345e5-b98b-4631-ac99-bafe79cccb05\" (UID: \"84c345e5-b98b-4631-ac99-bafe79cccb05\") " Nov 03 04:12:52.626710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626500 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:52.626710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626695 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:52.626833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626705 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:52.626833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.626531 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:12:52.627458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.627434 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c345e5-b98b-4631-ac99-bafe79cccb05-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.627566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.627463 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.627566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.627480 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.627566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.627494 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.627566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.627508 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.628910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.628884 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:12:52.629010 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.628944 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:52.629076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.629060 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:12:52.629120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.629078 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2" pod "84c345e5-b98b-4631-ac99-bafe79cccb05" (UID: "84c345e5-b98b-4631-ac99-bafe79cccb05"). InnerVolumeSpecName "kube-api-access-p4js2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:52.713082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.713046 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:52.729300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.729262 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.729300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.729301 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4js2\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-kube-api-access-p4js2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.729539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.729317 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:52.729539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:52.729331 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c345e5-b98b-4631-ac99-bafe79cccb05-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:53.519190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:53.519155 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht" Nov 03 04:12:53.519661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:53.519486 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:53.520623 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:53.520601 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:12:53.755236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:53.755201 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:12:53.763351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:53.763312 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4fa79-openshift-default-b558b4c45-h7jht"] Nov 03 04:12:54.986513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:54.986475 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" path="/var/lib/kubelet/pods/84c345e5-b98b-4631-ac99-bafe79cccb05/volumes" Nov 03 04:12:58.948914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.948881 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:12:58.949592 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.949574 5275 cpu_manager.go:414] "containerMap: removing container" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:58.949665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.949593 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:58.949665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.949619 5275 cpu_manager.go:414] "containerMap: removing container" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:58.949665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.949627 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:58.949811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.949748 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:58.954747 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:58.954728 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.097336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.094187 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-255c49\"" Nov 03 04:12:59.124146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.103337 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-255c49-dockercfg-lt4gz\"" Nov 03 04:12:59.125401 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.125355 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.125714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.125695 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9kj\" (UniqueName: \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.128095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.128074 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:12:59.153602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.153563 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:12:59.155059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.155034 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c345e5-b98b-4631-ac99-bafe79cccb05" containerName="istio-proxy" Nov 03 04:12:59.159907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.159118 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.171800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.171770 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:12:59.227891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.227756 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdr4g\" (UniqueName: \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.228082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.227924 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.228082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.227996 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.228194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.228099 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9kj\" (UniqueName: \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.228750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.228727 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.292281 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.292246 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9kj\" (UniqueName: \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\") pod \"payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.329838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.329802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdr4g\" (UniqueName: \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.330049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.329910 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.330748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.330723 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.416413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.416378 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdr4g\" (UniqueName: \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\") pod \"payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.477596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.477561 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:12:59.568908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.568876 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:12:59.863615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.863584 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:12:59.864721 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:12:59.864694 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e0fa20_3c55_4f37_8805_35b2b011959c.slice/crio-23086bfe7f68a7641fecb34979a6cba9539dd5a24a92744d96697779aa926dc8 WatchSource:0}: Error finding container 23086bfe7f68a7641fecb34979a6cba9539dd5a24a92744d96697779aa926dc8: Status 404 returned error can't find the container with id 23086bfe7f68a7641fecb34979a6cba9539dd5a24a92744d96697779aa926dc8 Nov 03 04:12:59.974099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:12:59.974068 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:12:59.975980 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:12:59.975956 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb980e6e1_3a3b_4074_8a11_70a6eefaab8d.slice/crio-a38da8ae3e3fc467826cabebd976555c2f02bb49491d24e2b3f558579e93f53c WatchSource:0}: Error finding container a38da8ae3e3fc467826cabebd976555c2f02bb49491d24e2b3f558579e93f53c: Status 404 returned error can't find the container with id a38da8ae3e3fc467826cabebd976555c2f02bb49491d24e2b3f558579e93f53c Nov 03 04:13:00.575665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.575629 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" event={"ID":"b980e6e1-3a3b-4074-8a11-70a6eefaab8d","Type":"ContainerStarted","Data":"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b"} Nov 03 04:13:00.575901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.575690 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" event={"ID":"b980e6e1-3a3b-4074-8a11-70a6eefaab8d","Type":"ContainerStarted","Data":"a38da8ae3e3fc467826cabebd976555c2f02bb49491d24e2b3f558579e93f53c"} Nov 03 04:13:00.575901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.575719 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:13:00.577628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.577599 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" event={"ID":"94e0fa20-3c55-4f37-8805-35b2b011959c","Type":"ContainerStarted","Data":"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84"} Nov 03 04:13:00.577731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.577650 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" event={"ID":"94e0fa20-3c55-4f37-8805-35b2b011959c","Type":"ContainerStarted","Data":"23086bfe7f68a7641fecb34979a6cba9539dd5a24a92744d96697779aa926dc8"} Nov 03 04:13:00.577731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.577688 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:13:00.693753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.693692 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" podStartSLOduration=2.6936735609999998 podStartE2EDuration="2.693673561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:13:00.691418029 +0000 UTC m=+4271.844106450" watchObservedRunningTime="2026-11-03 04:13:00.693673561 +0000 UTC m=+4271.846361982" Nov 03 04:13:00.810384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:00.809507 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" podStartSLOduration=1.809478828 podStartE2EDuration="1.809478828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:13:00.808925126 +0000 UTC m=+4271.961613549" watchObservedRunningTime="2026-11-03 04:13:00.809478828 +0000 UTC m=+4271.962167252" Nov 03 04:13:02.331028 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:13:02.330990 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:13:07.609216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.609177 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n"] Nov 03 04:13:07.613645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.613628 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.634383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.634349 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts48s\" (UniqueName: \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.634573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.634543 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.634642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.634631 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.666683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.666643 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:13:07.673013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.672981 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n"] Nov 03 04:13:07.716296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.716257 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:13:07.716296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.716270 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:13:07.716524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.716277 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:13:07.736156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.736122 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.736306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.736235 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts48s\" (UniqueName: \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.736421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.736403 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.737098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.737077 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.738614 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.738593 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.797468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.797438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts48s\" (UniqueName: \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\") pod \"maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:07.928061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:07.927971 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:08.347909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:08.347876 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n"] Nov 03 04:13:08.348601 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:13:08.348569 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b59f5e_4b68_4b52_8004_c767e166ee06.slice/crio-e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b WatchSource:0}: Error finding container e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b: Status 404 returned error can't find the container with id e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b Nov 03 04:13:08.642483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:08.642394 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" event={"ID":"16b59f5e-4b68-4b52-8004-c767e166ee06","Type":"ContainerStarted","Data":"e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b"} Nov 03 04:13:09.651386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:09.651295 5275 generic.go:356] "Generic (PLEG): container finished" podID="16b59f5e-4b68-4b52-8004-c767e166ee06" containerID="2a3b16c1e62519b57cad9c37cb2744c685ce6d60b8d802468708beb06157b657" exitCode=0 Nov 03 04:13:09.651923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:09.651381 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" event={"ID":"16b59f5e-4b68-4b52-8004-c767e166ee06","Type":"ContainerDied","Data":"2a3b16c1e62519b57cad9c37cb2744c685ce6d60b8d802468708beb06157b657"} Nov 03 04:13:10.838988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.838958 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:10.858980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.858931 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq"] Nov 03 04:13:10.860186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.860155 5275 cpu_manager.go:414] "containerMap: removing container" podUID="16b59f5e-4b68-4b52-8004-c767e166ee06" containerName="revoke-keys" Nov 03 04:13:10.860186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.860186 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b59f5e-4b68-4b52-8004-c767e166ee06" containerName="revoke-keys" Nov 03 04:13:10.860439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.860425 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b59f5e-4b68-4b52-8004-c767e166ee06" containerName="revoke-keys" Nov 03 04:13:10.864531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.864508 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:10.879442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.879396 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:10.879626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.879498 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhts\" (UniqueName: \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:10.879626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.879540 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:10.919820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.919731 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq"] Nov 03 04:13:10.983462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.983423 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\" (UniqueName: \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\") pod \"16b59f5e-4b68-4b52-8004-c767e166ee06\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " Nov 03 04:13:10.984272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.984242 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\") pod \"16b59f5e-4b68-4b52-8004-c767e166ee06\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " Nov 03 04:13:10.984420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.984315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\") pod \"16b59f5e-4b68-4b52-8004-c767e166ee06\" (UID: \"16b59f5e-4b68-4b52-8004-c767e166ee06\") " Nov 03 04:13:10.995159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:10.995095 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s" pod "16b59f5e-4b68-4b52-8004-c767e166ee06" (UID: "16b59f5e-4b68-4b52-8004-c767e166ee06"). InnerVolumeSpecName "kube-api-access-ts48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:13:11.002301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.002264 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca" pod "16b59f5e-4b68-4b52-8004-c767e166ee06" (UID: "16b59f5e-4b68-4b52-8004-c767e166ee06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:13:11.007741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.006820 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.007741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.006980 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhts\" (UniqueName: \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.007741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.007038 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.007741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.007379 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts48s\" (UniqueName: \"kubernetes.io/projected/16b59f5e-4b68-4b52-8004-c767e166ee06-kube-api-access-ts48s\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:11.007741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.007399 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:11.008579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.008552 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.013367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.013338 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.021983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.021956 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca" pod "16b59f5e-4b68-4b52-8004-c767e166ee06" (UID: "16b59f5e-4b68-4b52-8004-c767e166ee06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:13:11.071999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.071966 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhts\" (UniqueName: \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\") pod \"maas-api-revoke-keys-e2e-inf-a-eed598-77tcq\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.108652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.108610 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b59f5e-4b68-4b52-8004-c767e166ee06-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:11.184541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.184446 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:11.592031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.591982 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq"] Nov 03 04:13:11.594227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.594111 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:13:11.594227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.594189 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:13:11.597779 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:13:11.597739 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739675e3_a7fd_462d_a9b6_2cd150d2ab8a.slice/crio-ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0 WatchSource:0}: Error finding container ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0: Status 404 returned error can't find the container with id ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0 Nov 03 04:13:11.711777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.711731 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" event={"ID":"739675e3-a7fd-462d-a9b6-2cd150d2ab8a","Type":"ContainerStarted","Data":"ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0"} Nov 03 04:13:11.713801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.713774 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" event={"ID":"16b59f5e-4b68-4b52-8004-c767e166ee06","Type":"ContainerDied","Data":"e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b"} Nov 03 04:13:11.714237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.714220 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3e9d074605fff4b936c29a1763145234132c0bcbdb907414be2f16231765e3b" Nov 03 04:13:11.714384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:11.714161 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n" Nov 03 04:13:12.726305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:12.726268 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" event={"ID":"739675e3-a7fd-462d-a9b6-2cd150d2ab8a","Type":"ContainerStarted","Data":"b76acd3c04a93d28303391dc938e01b87855c485bf2389ac82c48f3fbab72734"} Nov 03 04:13:12.848876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:12.848552 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" podStartSLOduration=2.848522799 podStartE2EDuration="2.848522799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:13:12.842655468 +0000 UTC m=+4283.995343905" watchObservedRunningTime="2026-11-03 04:13:12.848522799 +0000 UTC m=+4284.001211219" Nov 03 04:13:13.782883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:13.782719 5275 generic.go:356] "Generic (PLEG): container finished" podID="739675e3-a7fd-462d-a9b6-2cd150d2ab8a" containerID="b76acd3c04a93d28303391dc938e01b87855c485bf2389ac82c48f3fbab72734" exitCode=0 Nov 03 04:13:13.782883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:13.782817 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" event={"ID":"739675e3-a7fd-462d-a9b6-2cd150d2ab8a","Type":"ContainerDied","Data":"b76acd3c04a93d28303391dc938e01b87855c485bf2389ac82c48f3fbab72734"} Nov 03 04:13:14.095929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.095459 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:13:14.095929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.095583 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:14.248874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.247569 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:14.256873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.253049 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:14.316873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.316349 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:14.351927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:14.350988 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:15.165652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.165539 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 04:13:15.306916 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.305986 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:15.321257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.320707 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:15.331547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.331352 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 04:13:15.331547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.331422 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 04:13:15.389887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.337932 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 04:13:15.403459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.403384 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 04:13:15.413334 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.413216 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 04:13:15.424154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.422830 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:15.523174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.523140 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\") pod \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " Nov 03 04:13:15.523174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.523186 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\") pod \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " Nov 03 04:13:15.523423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.523239 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\" (UniqueName: \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\") pod \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\" (UID: \"739675e3-a7fd-462d-a9b6-2cd150d2ab8a\") " Nov 03 04:13:15.523913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.523877 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca" pod "739675e3-a7fd-462d-a9b6-2cd150d2ab8a" (UID: "739675e3-a7fd-462d-a9b6-2cd150d2ab8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:13:15.525741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.525709 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca" pod "739675e3-a7fd-462d-a9b6-2cd150d2ab8a" (UID: "739675e3-a7fd-462d-a9b6-2cd150d2ab8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:13:15.525964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.525939 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts" pod "739675e3-a7fd-462d-a9b6-2cd150d2ab8a" (UID: "739675e3-a7fd-462d-a9b6-2cd150d2ab8a"). InnerVolumeSpecName "kube-api-access-nbhts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:13:15.625035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.625000 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:15.625035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.625040 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:15.625241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.625055 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhts\" (UniqueName: \"kubernetes.io/projected/739675e3-a7fd-462d-a9b6-2cd150d2ab8a-kube-api-access-nbhts\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:15.892724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.892667 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" event={"ID":"739675e3-a7fd-462d-a9b6-2cd150d2ab8a","Type":"ContainerDied","Data":"ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0"} Nov 03 04:13:15.892724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.892731 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8e34a4438c2e4bb3d952c14b19b799d6f590138365befad12eea721f11d8c0" Nov 03 04:13:15.893080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:15.892785 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq" Nov 03 04:13:22.377727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.377054 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": read tcp 10.133.0.2:46482->10.133.1.33:15021: read: connection reset by peer" start-of-body= Nov 03 04:13:22.377727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.377112 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": read tcp 10.133.0.2:46482->10.133.1.33:15021: read: connection reset by peer" Nov 03 04:13:22.955296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.955254 5275 generic.go:356] "Generic (PLEG): container finished" podID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerID="a18639815908874aa320ef5ae8843367e37fc745547e534af3cfb012d342d6a6" exitCode=0 Nov 03 04:13:22.955874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.955821 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerDied","Data":"a18639815908874aa320ef5ae8843367e37fc745547e534af3cfb012d342d6a6"} Nov 03 04:13:22.956729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.956703 5275 scope.go:122] "RemoveContainer" containerID="a18639815908874aa320ef5ae8843367e37fc745547e534af3cfb012d342d6a6" Nov 03 04:13:22.958653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.958622 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:13:22.958745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.958695 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:13:22.958745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:22.958730 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:13:23.978614 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:23.978580 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerStarted","Data":"71adf17ecef6fb03acb718447527c89ab05e0064223eba97c3371de813669eaf"} Nov 03 04:13:24.570873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:24.570259 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:13:24.570873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:24.570395 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 03 04:13:24.570873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:24.570439 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 03 04:13:25.570343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:25.570308 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 03 04:13:25.571001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:25.570965 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 03 04:13:26.567413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:26.567377 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 03 04:13:26.567750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:26.567705 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 03 04:13:27.569872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:27.569015 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 03 04:13:27.569872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:27.569094 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 03 04:13:28.574327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:28.574280 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 03 04:13:28.574762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:28.574351 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 03 04:13:29.569290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:29.569254 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:13:29.569760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:29.569742 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:13:29.570462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:29.570445 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:13:32.334594 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:13:32.334558 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:13:50.421410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.421366 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5"] Nov 03 04:13:50.422128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.422111 5275 cpu_manager.go:414] "containerMap: removing container" podUID="739675e3-a7fd-462d-a9b6-2cd150d2ab8a" containerName="revoke-keys" Nov 03 04:13:50.422184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.422129 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739675e3-a7fd-462d-a9b6-2cd150d2ab8a" containerName="revoke-keys" Nov 03 04:13:50.422242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.422232 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="739675e3-a7fd-462d-a9b6-2cd150d2ab8a" containerName="revoke-keys" Nov 03 04:13:50.424716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.424697 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.479458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.479417 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:13:50.525155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.525118 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.525337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.525305 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.525431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.525419 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7zh\" (UniqueName: \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.529083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.529052 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:13:50.529083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.529070 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:13:50.529261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.529195 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:13:50.532529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.532507 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5"] Nov 03 04:13:50.627099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.627063 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7zh\" (UniqueName: \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.627311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.627134 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.627311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.627282 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.627952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.627930 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.629706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.629688 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.688907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.688812 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7zh\" (UniqueName: \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\") pod \"maas-api-revoke-sub-2ba70333-qwkq5\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:50.738949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:50.738916 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:51.130997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:51.130963 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5"] Nov 03 04:13:51.131702 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:13:51.131677 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28614d35_5281_42d3_a4e8_9e743ad301cb.slice/crio-002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902 WatchSource:0}: Error finding container 002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902: Status 404 returned error can't find the container with id 002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902 Nov 03 04:13:51.198976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:51.198945 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" event={"ID":"28614d35-5281-42d3-a4e8-9e743ad301cb","Type":"ContainerStarted","Data":"002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902"} Nov 03 04:13:52.207841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:52.207808 5275 generic.go:356] "Generic (PLEG): container finished" podID="28614d35-5281-42d3-a4e8-9e743ad301cb" containerID="de967cdb3a5d9aee83818f2cb34cd98715e72921359633a8c89861c9c8d0696b" exitCode=0 Nov 03 04:13:52.208307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:52.207896 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" event={"ID":"28614d35-5281-42d3-a4e8-9e743ad301cb","Type":"ContainerDied","Data":"de967cdb3a5d9aee83818f2cb34cd98715e72921359633a8c89861c9c8d0696b"} Nov 03 04:13:53.368955 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.368926 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:53.462922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.462885 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\") pod \"28614d35-5281-42d3-a4e8-9e743ad301cb\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " Nov 03 04:13:53.463140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.463095 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\" (UniqueName: \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\") pod \"28614d35-5281-42d3-a4e8-9e743ad301cb\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " Nov 03 04:13:53.463205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.463138 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\") pod \"28614d35-5281-42d3-a4e8-9e743ad301cb\" (UID: \"28614d35-5281-42d3-a4e8-9e743ad301cb\") " Nov 03 04:13:53.463646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.463612 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca" pod "28614d35-5281-42d3-a4e8-9e743ad301cb" (UID: "28614d35-5281-42d3-a4e8-9e743ad301cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:13:53.465266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.465236 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca" pod "28614d35-5281-42d3-a4e8-9e743ad301cb" (UID: "28614d35-5281-42d3-a4e8-9e743ad301cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:13:53.465767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.465720 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh" pod "28614d35-5281-42d3-a4e8-9e743ad301cb" (UID: "28614d35-5281-42d3-a4e8-9e743ad301cb"). InnerVolumeSpecName "kube-api-access-ql7zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:13:53.564839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.564795 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:53.565028 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.564840 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7zh\" (UniqueName: \"kubernetes.io/projected/28614d35-5281-42d3-a4e8-9e743ad301cb-kube-api-access-ql7zh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:53.565028 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.564875 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28614d35-5281-42d3-a4e8-9e743ad301cb-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:13:53.868249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.868210 5275 scope.go:122] "RemoveContainer" containerID="c77500aa970d11c6963a195e017ef9f7494070be08bfd3302a950f2fa59bb130" Nov 03 04:13:53.879338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:53.879309 5275 scope.go:122] "RemoveContainer" containerID="d4e60c428ba8eea9d930479e88ce0ff007225f4673db6e44c415ca6eb884b187" Nov 03 04:13:54.224076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:54.223988 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" Nov 03 04:13:54.224076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:54.224017 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5" event={"ID":"28614d35-5281-42d3-a4e8-9e743ad301cb","Type":"ContainerDied","Data":"002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902"} Nov 03 04:13:54.224076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:13:54.224059 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="002273af40a72f45024557cdcf225d4533be3cd397e0753ff45c4c9bb1832902" Nov 03 04:14:02.329831 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:14:02.329796 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:14:06.044023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.043982 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:14:06.044451 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.044200 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" containerID="cri-o://1a33b14b3e226c7bc93e89e6a797f23fe984df0bbdb26d296f66e22acb5bd49d" gracePeriod=30 Nov 03 04:14:06.192615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.192577 5275 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5t4rh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.29:8080/ready\": dial tcp 10.133.1.29:8080: connect: connection refused" start-of-body= Nov 03 04:14:06.192791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.192663 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.1.29:8080/ready\": dial tcp 10.133.1.29:8080: connect: connection refused" Nov 03 04:14:06.322945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.322907 5275 generic.go:356] "Generic (PLEG): container finished" podID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerID="1a33b14b3e226c7bc93e89e6a797f23fe984df0bbdb26d296f66e22acb5bd49d" exitCode=0 Nov 03 04:14:06.323129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.322949 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" event={"ID":"0a277337-efd5-4e77-9ffa-11da1a86461a","Type":"ContainerDied","Data":"1a33b14b3e226c7bc93e89e6a797f23fe984df0bbdb26d296f66e22acb5bd49d"} Nov 03 04:14:06.323129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.323010 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" event={"ID":"0a277337-efd5-4e77-9ffa-11da1a86461a","Type":"ContainerDied","Data":"8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c"} Nov 03 04:14:06.323129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.323029 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df874a539c4e29d8e69fd5c1b715dcdf6938a8a4374eb23c8236c8f8388cc3c" Nov 03 04:14:06.332036 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.332006 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:14:06.439136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439106 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439162 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439193 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439310 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439473 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439509 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.439666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.439592 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\") pod \"0a277337-efd5-4e77-9ffa-11da1a86461a\" (UID: \"0a277337-efd5-4e77-9ffa-11da1a86461a\") " Nov 03 04:14:06.440289 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.440234 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:14:06.442283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.442254 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:14:06.442417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.442316 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:14:06.442417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.442325 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:14:06.443223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.443199 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:14:06.443223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.443209 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:14:06.443475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.443457 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml" pod "0a277337-efd5-4e77-9ffa-11da1a86461a" (UID: "0a277337-efd5-4e77-9ffa-11da1a86461a"). InnerVolumeSpecName "kube-api-access-6xhml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:14:06.540988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.540959 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-ca-configmap\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.540988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.540985 5275 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a277337-efd5-4e77-9ffa-11da1a86461a-local-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.540988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.540995 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhml\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-kube-api-access-6xhml\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.541235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.541004 5275 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-kubeconfig\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.541235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.541013 5275 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-cacerts\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.541235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.541022 5275 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-csr-dns-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:06.541235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:06.541031 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a277337-efd5-4e77-9ffa-11da1a86461a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:07.347336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:07.345931 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh" Nov 03 04:14:07.573356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:07.573316 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:14:07.581254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:07.581217 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5t4rh"] Nov 03 04:14:08.985655 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:08.985618 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" path="/var/lib/kubelet/pods/0a277337-efd5-4e77-9ffa-11da1a86461a/volumes" Nov 03 04:14:13.048486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:13.048445 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:14:13.152455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:13.152406 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:14:18.050085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.050042 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" podUID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" containerName="payload-processing" containerID="cri-o://4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e" gracePeriod=25 Nov 03 04:14:18.153148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.153107 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" podUID="740ab299-86ab-411f-a5b9-a3a94ee2da26" containerName="payload-pre-processing" containerID="cri-o://ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a" gracePeriod=25 Nov 03 04:14:18.341814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.341784 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:14:18.459325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.459300 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\" (UniqueName: \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\") pod \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " Nov 03 04:14:18.459504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.459458 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\") pod \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\" (UID: \"6cf88a52-6914-45c8-bee4-80b7e85d4ecc\") " Nov 03 04:14:18.460287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.460076 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume" pod "6cf88a52-6914-45c8-bee4-80b7e85d4ecc" (UID: "6cf88a52-6914-45c8-bee4-80b7e85d4ecc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:14:18.464567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.464510 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5" pod "6cf88a52-6914-45c8-bee4-80b7e85d4ecc" (UID: "6cf88a52-6914-45c8-bee4-80b7e85d4ecc"). InnerVolumeSpecName "kube-api-access-vnxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:14:18.467341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.467323 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:14:18.472250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.472228 5275 generic.go:356] "Generic (PLEG): container finished" podID="740ab299-86ab-411f-a5b9-a3a94ee2da26" containerID="ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a" exitCode=0 Nov 03 04:14:18.472365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.472266 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" Nov 03 04:14:18.472365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.472313 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" event={"ID":"740ab299-86ab-411f-a5b9-a3a94ee2da26","Type":"ContainerDied","Data":"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a"} Nov 03 04:14:18.472365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.472360 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5" event={"ID":"740ab299-86ab-411f-a5b9-a3a94ee2da26","Type":"ContainerDied","Data":"0424600cd2b0b1c46ec0a71c60c16a61c66d02e137556d16436eb85e944a48fe"} Nov 03 04:14:18.472517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.472386 5275 scope.go:122] "RemoveContainer" containerID="ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a" Nov 03 04:14:18.474106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.474037 5275 generic.go:356] "Generic (PLEG): container finished" podID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" containerID="4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e" exitCode=0 Nov 03 04:14:18.474334 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.474305 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" event={"ID":"6cf88a52-6914-45c8-bee4-80b7e85d4ecc","Type":"ContainerDied","Data":"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e"} Nov 03 04:14:18.474434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.474366 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" event={"ID":"6cf88a52-6914-45c8-bee4-80b7e85d4ecc","Type":"ContainerDied","Data":"21bd35ba7603bd5366e63f4a69454d7d817edfd3c8f38b81f83124f1caf68a90"} Nov 03 04:14:18.474477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.474435 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh" Nov 03 04:14:18.494138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.494073 5275 scope.go:122] "RemoveContainer" containerID="ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a" Nov 03 04:14:18.494491 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:14:18.494468 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a\": container with ID starting with ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a not found: ID does not exist" containerID="ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a" Nov 03 04:14:18.494552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.494500 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a"} err="failed to get container status \"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a\": rpc error: code = NotFound desc = could not find container \"ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a\": container with ID starting with ea3edf965c1b16e455e965a95288f3f079ba450d8b0d94bed9599cafced0a10a not found: ID does not exist" Nov 03 04:14:18.494552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.494519 5275 scope.go:122] "RemoveContainer" containerID="4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e" Nov 03 04:14:18.507692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.507666 5275 scope.go:122] "RemoveContainer" containerID="4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e" Nov 03 04:14:18.508088 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:14:18.508054 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e\": container with ID starting with 4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e not found: ID does not exist" containerID="4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e" Nov 03 04:14:18.508195 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.508096 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e"} err="failed to get container status \"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e\": rpc error: code = NotFound desc = could not find container \"4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e\": container with ID starting with 4700d2e140f1f9e7db04feb520d5b5cda758c46229a538f83156d1eac4c9191e not found: ID does not exist" Nov 03 04:14:18.560639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.560606 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\") pod \"740ab299-86ab-411f-a5b9-a3a94ee2da26\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " Nov 03 04:14:18.560882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.560732 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\" (UniqueName: \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\") pod \"740ab299-86ab-411f-a5b9-a3a94ee2da26\" (UID: \"740ab299-86ab-411f-a5b9-a3a94ee2da26\") " Nov 03 04:14:18.561124 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.561096 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume" pod "740ab299-86ab-411f-a5b9-a3a94ee2da26" (UID: "740ab299-86ab-411f-a5b9-a3a94ee2da26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:14:18.563390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.563363 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:18.563390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.563392 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxk5\" (UniqueName: \"kubernetes.io/projected/6cf88a52-6914-45c8-bee4-80b7e85d4ecc-kube-api-access-vnxk5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:18.563390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.563414 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ab299-86ab-411f-a5b9-a3a94ee2da26-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:18.563905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.563831 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz" pod "740ab299-86ab-411f-a5b9-a3a94ee2da26" (UID: "740ab299-86ab-411f-a5b9-a3a94ee2da26"). InnerVolumeSpecName "kube-api-access-zk6tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:14:18.665276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.665191 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6tz\" (UniqueName: \"kubernetes.io/projected/740ab299-86ab-411f-a5b9-a3a94ee2da26-kube-api-access-zk6tz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:18.696951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.696912 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:14:18.700496 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.700467 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ed4ecc-6d78b9888f-z9wxh"] Nov 03 04:14:18.901736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.901685 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:14:18.904670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.904643 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ed4ecc-76886d985f-78vg5"] Nov 03 04:14:18.987165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.987075 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" path="/var/lib/kubelet/pods/6cf88a52-6914-45c8-bee4-80b7e85d4ecc/volumes" Nov 03 04:14:18.987562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:18.987544 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740ab299-86ab-411f-a5b9-a3a94ee2da26" path="/var/lib/kubelet/pods/740ab299-86ab-411f-a5b9-a3a94ee2da26/volumes" Nov 03 04:14:32.329923 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:14:32.329886 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:14:34.616121 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:34.616081 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n"] Nov 03 04:14:34.665110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:34.665072 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-934bd4-mtx9n"] Nov 03 04:14:34.985448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:34.985374 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b59f5e-4b68-4b52-8004-c767e166ee06" path="/var/lib/kubelet/pods/16b59f5e-4b68-4b52-8004-c767e166ee06/volumes" Nov 03 04:14:40.899253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:40.899220 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq"] Nov 03 04:14:40.948470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:40.948434 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eed598-77tcq"] Nov 03 04:14:40.993243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:40.993209 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739675e3-a7fd-462d-a9b6-2cd150d2ab8a" path="/var/lib/kubelet/pods/739675e3-a7fd-462d-a9b6-2cd150d2ab8a/volumes" Nov 03 04:14:44.696504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.696360 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:14:44.698348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698321 5275 cpu_manager.go:414] "containerMap: removing container" podUID="740ab299-86ab-411f-a5b9-a3a94ee2da26" containerName="payload-pre-processing" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698350 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740ab299-86ab-411f-a5b9-a3a94ee2da26" containerName="payload-pre-processing" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698367 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698374 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698424 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" containerName="payload-processing" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698432 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" containerName="payload-processing" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698443 5275 cpu_manager.go:414] "containerMap: removing container" podUID="28614d35-5281-42d3-a4e8-9e743ad301cb" containerName="revoke-keys" Nov 03 04:14:44.698458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698450 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28614d35-5281-42d3-a4e8-9e743ad301cb" containerName="revoke-keys" Nov 03 04:14:44.698749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698640 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf88a52-6914-45c8-bee4-80b7e85d4ecc" containerName="payload-processing" Nov 03 04:14:44.698749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698654 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="28614d35-5281-42d3-a4e8-9e743ad301cb" containerName="revoke-keys" Nov 03 04:14:44.698749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698673 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="740ab299-86ab-411f-a5b9-a3a94ee2da26" containerName="payload-pre-processing" Nov 03 04:14:44.698749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.698682 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a277337-efd5-4e77-9ffa-11da1a86461a" containerName="discovery" Nov 03 04:14:44.730386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.730354 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.763928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.763748 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.764384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.764356 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.764662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.764635 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.764879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.764835 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.765169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.765145 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.765379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.765359 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw742\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.765586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.765568 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.765804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.765788 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.766050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.766031 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.813778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.813710 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e30496c2-openshift-default-dockercfg-68wtb\"" Nov 03 04:14:44.869877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869640 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.869877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869719 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.869877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.869877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869829 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.870266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869895 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.870266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.869927 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.870266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.870039 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.870266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.870106 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw742\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.870266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.870137 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.873552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.871026 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.873552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.871085 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:14:44.873552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.871408 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.873552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.871648 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.873552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.872288 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.874066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.873623 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.906898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.903431 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.907626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.907365 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.933328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.933284 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:44.933493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:44.933375 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw742\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\") pod \"e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:45.137310 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.137275 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:45.599990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.599959 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:14:45.602473 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:14:45.602448 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd544c042_ccbf_4de9_bce4_50bdcd8c4de1.slice/crio-13e1382eaf63f34a81febf1a869dfcc33673608aeeb8aa87462f72364415d2ed WatchSource:0}: Error finding container 13e1382eaf63f34a81febf1a869dfcc33673608aeeb8aa87462f72364415d2ed: Status 404 returned error can't find the container with id 13e1382eaf63f34a81febf1a869dfcc33673608aeeb8aa87462f72364415d2ed Nov 03 04:14:45.605296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.605261 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:45.605377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.605328 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:45.605377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.605358 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:45.707478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:45.707441 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerStarted","Data":"13e1382eaf63f34a81febf1a869dfcc33673608aeeb8aa87462f72364415d2ed"} Nov 03 04:14:46.717388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.717342 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerStarted","Data":"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50"} Nov 03 04:14:46.719294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.719267 5275 generic.go:356] "Generic (PLEG): container finished" podID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerID="79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6" exitCode=0 Nov 03 04:14:46.719442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.719345 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerDied","Data":"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6"} Nov 03 04:14:46.719710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.719695 5275 scope.go:122] "RemoveContainer" containerID="79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6" Nov 03 04:14:46.721648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.721614 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:46.721738 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.721709 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:46.721779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.721760 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:14:46.927311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:46.927252 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podStartSLOduration=2.927236218 podStartE2EDuration="2.927236218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:14:46.886564993 +0000 UTC m=+4378.039253415" watchObservedRunningTime="2026-11-03 04:14:46.927236218 +0000 UTC m=+4378.079924639" Nov 03 04:14:47.137658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:47.137626 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:47.143805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:47.143773 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:47.728716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:47.728677 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerStarted","Data":"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7"} Nov 03 04:14:47.729252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:47.729141 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:47.730167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:47.730138 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:14:48.705524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:48.705474 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:14:48.705753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:48.705691 5275 patch_prober.go:29] interesting pod/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 03 04:14:48.705824 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:48.705753 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 03 04:14:49.705839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:49.705792 5275 patch_prober.go:29] interesting pod/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 03 04:14:49.706395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:49.705888 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 03 04:14:50.706083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:50.706042 5275 patch_prober.go:29] interesting pod/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 03 04:14:50.706583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:50.706119 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 03 04:14:51.705591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:51.705550 5275 patch_prober.go:29] interesting pod/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 03 04:14:51.705789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:51.705618 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 03 04:14:52.706018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:52.705985 5275 patch_prober.go:29] interesting pod/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 03 04:14:52.706435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:52.706038 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 03 04:14:53.519743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.519705 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:14:53.533326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.533297 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:14:53.536355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.536339 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.561122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.561090 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.561314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.561178 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfms\" (UniqueName: \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.594344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.594304 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:14:53.653873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.653158 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e30496c2-dockercfg-4dhtg\"" Nov 03 04:14:53.653873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.653233 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e30496c2\"" Nov 03 04:14:53.675763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.675717 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfms\" (UniqueName: \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.679430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.679387 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.680745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.680713 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.701895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.701834 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:14:53.723898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.712683 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.723898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.723751 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:14:53.742217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.742190 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfms\" (UniqueName: \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\") pod \"payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.752115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.752086 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:14:53.780841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.780811 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.781054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.780967 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7lq\" (UniqueName: \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.810284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.810247 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:14:53.849546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.849502 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:53.882869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.882818 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.883055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.882961 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7lq\" (UniqueName: \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.884086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.884054 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:53.914025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.913991 5275 scope.go:122] "RemoveContainer" containerID="1a33b14b3e226c7bc93e89e6a797f23fe984df0bbdb26d296f66e22acb5bd49d" Nov 03 04:14:53.947450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:53.947424 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7lq\" (UniqueName: \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\") pod \"payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:54.045048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.044971 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:54.236053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.236023 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:14:54.237187 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:14:54.237159 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbc470b_839c_4a3d_a1c0_c1679e0aab78.slice/crio-2202209bfb91457b1eb244940a6da772a35c368047469f4f764ecbc3ebb9f299 WatchSource:0}: Error finding container 2202209bfb91457b1eb244940a6da772a35c368047469f4f764ecbc3ebb9f299: Status 404 returned error can't find the container with id 2202209bfb91457b1eb244940a6da772a35c368047469f4f764ecbc3ebb9f299 Nov 03 04:14:54.431127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.431098 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:14:54.432029 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:14:54.432001 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935196d0_bd74_4d5d_addd_99261d675799.slice/crio-66bdb14df407602b7395c6d5c2830be66b6f81e70f7bf10ae98006ec4c706b6b WatchSource:0}: Error finding container 66bdb14df407602b7395c6d5c2830be66b6f81e70f7bf10ae98006ec4c706b6b: Status 404 returned error can't find the container with id 66bdb14df407602b7395c6d5c2830be66b6f81e70f7bf10ae98006ec4c706b6b Nov 03 04:14:54.818487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.818447 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" event={"ID":"935196d0-bd74-4d5d-addd-99261d675799","Type":"ContainerStarted","Data":"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371"} Nov 03 04:14:54.818996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.818498 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" event={"ID":"935196d0-bd74-4d5d-addd-99261d675799","Type":"ContainerStarted","Data":"66bdb14df407602b7395c6d5c2830be66b6f81e70f7bf10ae98006ec4c706b6b"} Nov 03 04:14:54.818996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.818647 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:14:54.824520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.824479 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" event={"ID":"fbbc470b-839c-4a3d-a1c0-c1679e0aab78","Type":"ContainerStarted","Data":"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5"} Nov 03 04:14:54.824701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.824544 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" event={"ID":"fbbc470b-839c-4a3d-a1c0-c1679e0aab78","Type":"ContainerStarted","Data":"2202209bfb91457b1eb244940a6da772a35c368047469f4f764ecbc3ebb9f299"} Nov 03 04:14:54.824701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.824575 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:14:54.939264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:54.939199 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" podStartSLOduration=1.939179284 podStartE2EDuration="1.939179284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:14:54.935582871 +0000 UTC m=+4386.088271294" watchObservedRunningTime="2026-11-03 04:14:54.939179284 +0000 UTC m=+4386.091867709" Nov 03 04:14:55.055989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:14:55.055934 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" podStartSLOduration=2.055918157 podStartE2EDuration="2.055918157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:14:55.051019529 +0000 UTC m=+4386.203707956" watchObservedRunningTime="2026-11-03 04:14:55.055918157 +0000 UTC m=+4386.208606583" Nov 03 04:15:00.187271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.187232 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:00.191696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.191672 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:00.191830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.191813 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.195548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.195525 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:00.195663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.195636 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.199146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.199122 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.236355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.236315 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:00.239587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.239560 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.239725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.239664 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:00.242814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.242791 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.242814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.242803 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:00.246808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.246784 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.247764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.247118 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:00.250438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.250394 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.250865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.250828 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:00.253585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.253566 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.253703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.253640 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:00.257031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.257009 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:00.257276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.257243 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.260775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.260756 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273175 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldng\" (UniqueName: \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273323 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273376 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7bg\" (UniqueName: \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273439 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273585 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.274661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.273615 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscv7\" (UniqueName: \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.282267 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.282245 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:15:00.282393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.282266 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:15:00.282393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.282338 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:15:00.287903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.287878 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:00.291305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.291284 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:00.291443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.291369 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.328656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.328618 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:00.335388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.335360 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:00.347577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.347546 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:00.349683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.349642 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:00.354473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.354452 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:00.382164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.382113 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j987\" (UniqueName: \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.382418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.382397 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4r9\" (UniqueName: \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.382567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.382551 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb57z\" (UniqueName: \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.383564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383535 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.383704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383582 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.383704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383625 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bsw\" (UniqueName: \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.383704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383671 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.383927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383718 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.383927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383756 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.383927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383793 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.383927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383841 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldng\" (UniqueName: \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.383927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383924 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.384151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.383966 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.384151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384010 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85bs\" (UniqueName: \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.384151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384055 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.384151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384107 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7rg\" (UniqueName: \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.384151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384145 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7bg\" (UniqueName: \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.384333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384190 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.384333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384323 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkqw\" (UniqueName: \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.384424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384407 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.384479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384463 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscv7\" (UniqueName: \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.384536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.384507 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnwr\" (UniqueName: \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.385978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.385953 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.387103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.387080 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.387251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.387109 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.392566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.387367 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:00.397865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.393612 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:00.407637 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.407605 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:00.436067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.436031 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:00.457184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.457093 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:00.467074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.467040 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldng\" (UniqueName: \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.467308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.467286 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7bg\" (UniqueName: \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.485889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.485811 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscv7\" (UniqueName: \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.486223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486193 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.486321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486241 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.486321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486276 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bsw\" (UniqueName: \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.486321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486314 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.486500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486354 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.486500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486384 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.486500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486411 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.486500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486474 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.486719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486507 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.486719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486545 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85bs\" (UniqueName: \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.486719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486597 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7rg\" (UniqueName: \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.486719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486707 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.486719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486711 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkqw\" (UniqueName: \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.487078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486790 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnwr\" (UniqueName: \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.487078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486834 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j987\" (UniqueName: \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.487078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486872 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4r9\" (UniqueName: \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.487078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.486900 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb57z\" (UniqueName: \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.487765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.487733 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.487996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.487977 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.488382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.488360 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.488465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.488396 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.488837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.488819 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.489314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.489295 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.489404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.489347 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.509294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.509264 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:00.517262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.517239 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:00.524583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.524561 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:00.558127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.558036 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnwr\" (UniqueName: \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.558427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.558396 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j987\" (UniqueName: \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\") pod \"maas-api-key-cleanup-29894655-5szgz\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.561956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.561901 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkqw\" (UniqueName: \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.562546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.562525 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85bs\" (UniqueName: \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\") pod \"maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:00.562680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.562661 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4r9\" (UniqueName: \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.562974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.562950 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bsw\" (UniqueName: \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.563601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.563574 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7rg\" (UniqueName: \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\") pod \"maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.564383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.564359 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb57z\" (UniqueName: \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.567486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.567279 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:00.574678 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.573976 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:00.585728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.585493 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:00.615596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.592244 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:00.615596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.601516 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:00.615596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.608767 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:00.619510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.617139 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:00.858282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:00.857817 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:01.108327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.108283 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:01.109404 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.109377 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1452961_d9b1_4411_ad3b_74b96983de5f.slice/crio-ff431d3bce589ce2dbe6a7da120b2f3f411fc5f91bc48e6ed3715f09f93dd52f WatchSource:0}: Error finding container ff431d3bce589ce2dbe6a7da120b2f3f411fc5f91bc48e6ed3715f09f93dd52f: Status 404 returned error can't find the container with id ff431d3bce589ce2dbe6a7da120b2f3f411fc5f91bc48e6ed3715f09f93dd52f Nov 03 04:15:01.130631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.130530 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:01.131001 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.130966 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c75986_6ac3_456f_b746_6fc49e78f9d5.slice/crio-b5b9f4ade6cf45079610c966f15925d05add6e0ceb3892c019bbf3592339dee6 WatchSource:0}: Error finding container b5b9f4ade6cf45079610c966f15925d05add6e0ceb3892c019bbf3592339dee6: Status 404 returned error can't find the container with id b5b9f4ade6cf45079610c966f15925d05add6e0ceb3892c019bbf3592339dee6 Nov 03 04:15:01.157871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.157825 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:01.174388 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.174360 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf644c466_1b9c_495d_b9d8_193c22509b2e.slice/crio-5a6161151721b65f9904ece7675b096dad235f792e4094951470954f928c4f4a WatchSource:0}: Error finding container 5a6161151721b65f9904ece7675b096dad235f792e4094951470954f928c4f4a: Status 404 returned error can't find the container with id 5a6161151721b65f9904ece7675b096dad235f792e4094951470954f928c4f4a Nov 03 04:15:01.225472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.225443 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:01.226036 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.226008 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c01e88_5156_4a40_96bc_727cedb46c81.slice/crio-cab37b35625ed72e4b1391bccd687c8b6ece683a321c23674bd6249b99d65b42 WatchSource:0}: Error finding container cab37b35625ed72e4b1391bccd687c8b6ece683a321c23674bd6249b99d65b42: Status 404 returned error can't find the container with id cab37b35625ed72e4b1391bccd687c8b6ece683a321c23674bd6249b99d65b42 Nov 03 04:15:01.251691 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.249709 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:01.255893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.255826 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:01.260865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.260557 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:01.261380 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.261355 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c94ec6_28aa_4898_8b15_3d13e20b05e2.slice/crio-d59193714d21ad5e6234288d8e1d07c49eb46dc4cae43437eadab6e76bbe3662 WatchSource:0}: Error finding container d59193714d21ad5e6234288d8e1d07c49eb46dc4cae43437eadab6e76bbe3662: Status 404 returned error can't find the container with id d59193714d21ad5e6234288d8e1d07c49eb46dc4cae43437eadab6e76bbe3662 Nov 03 04:15:01.316943 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.316915 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:01.317578 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.317556 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fd1bf8_d2a5_4151_ad29_8cf28dcf5000.slice/crio-e09a747df14b7a512ee1fb056111b5067f0dffec2a8b7fc716a2483f99c43226 WatchSource:0}: Error finding container e09a747df14b7a512ee1fb056111b5067f0dffec2a8b7fc716a2483f99c43226: Status 404 returned error can't find the container with id e09a747df14b7a512ee1fb056111b5067f0dffec2a8b7fc716a2483f99c43226 Nov 03 04:15:01.355131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.354655 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:01.360610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.359941 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:01.361935 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:01.361910 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f63c44a_1883_4b56_b061_4ec2128a1ea2.slice/crio-e011f5ff931068354f7bcf53f1d9a9938a7fa3cb4a3081a0b98b8380cb9ce9c7 WatchSource:0}: Error finding container e011f5ff931068354f7bcf53f1d9a9938a7fa3cb4a3081a0b98b8380cb9ce9c7: Status 404 returned error can't find the container with id e011f5ff931068354f7bcf53f1d9a9938a7fa3cb4a3081a0b98b8380cb9ce9c7 Nov 03 04:15:01.365612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.365519 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:01.884424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.884390 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerStarted","Data":"f9a8c6913fd531e846cf255eb0dbb7803cc4ecfbabe6d6ff3993e841d7934c69"} Nov 03 04:15:01.884605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.884473 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerStarted","Data":"e09a747df14b7a512ee1fb056111b5067f0dffec2a8b7fc716a2483f99c43226"} Nov 03 04:15:01.886087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.886060 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerStarted","Data":"f673150a7a1a20e047d9613a520ed3c3eb51ba7942113c1ee0df49d39221a9a9"} Nov 03 04:15:01.886196 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.886101 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerStarted","Data":"cab37b35625ed72e4b1391bccd687c8b6ece683a321c23674bd6249b99d65b42"} Nov 03 04:15:01.890442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.890417 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerStarted","Data":"3efce2cbfbc3f88acb4151d7b3f7d09a7f5d962a821a0d993cefafdfd34ec54a"} Nov 03 04:15:01.890546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.890462 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerStarted","Data":"5a6161151721b65f9904ece7675b096dad235f792e4094951470954f928c4f4a"} Nov 03 04:15:01.892164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.892144 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerStarted","Data":"7f5f951bad78bda5f5d38fe44d587a2a31c809120a52737f5d478380e6361a5c"} Nov 03 04:15:01.892266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.892189 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerStarted","Data":"be89a086242fdf6efcbd70a48bae44107eba11cd34462fc162f574a987093c84"} Nov 03 04:15:01.893538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.893509 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerStarted","Data":"e011f5ff931068354f7bcf53f1d9a9938a7fa3cb4a3081a0b98b8380cb9ce9c7"} Nov 03 04:15:01.895394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.895374 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerStarted","Data":"517c02293fdce3871cbe61fde0471be30461e29437fc64abdedeae81459ff122"} Nov 03 04:15:01.895502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.895410 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerStarted","Data":"ff431d3bce589ce2dbe6a7da120b2f3f411fc5f91bc48e6ed3715f09f93dd52f"} Nov 03 04:15:01.897381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.897348 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerStarted","Data":"40d2a2963a3cb1b773e754fd589f1a984a0b3f60071356574f43cf1235898c4d"} Nov 03 04:15:01.897560 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.897390 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerStarted","Data":"b219ee6a4e313131b7af41349fe1b48f15792965d7942568745b349bece64db6"} Nov 03 04:15:01.899176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.899156 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerStarted","Data":"ad5dc7c072a3ec9b77eb6ef62f608a4f8558af1e7101cba8d1991370e23b6565"} Nov 03 04:15:01.899271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.899194 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerStarted","Data":"c197fcf403f86df83b9244ec541c42c2db25b79cc7ab177223d09a0f579dae19"} Nov 03 04:15:01.900687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.900669 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerStarted","Data":"3b83bf4cfcc8d5040a8803a6a175a4e2d99d8ac9b81c00554a488a1d833d04a9"} Nov 03 04:15:01.900796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.900703 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerStarted","Data":"d59193714d21ad5e6234288d8e1d07c49eb46dc4cae43437eadab6e76bbe3662"} Nov 03 04:15:01.901939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.901918 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerStarted","Data":"0893cc5c4d4e85debd6908dde0ac3be82e519075f096cc49d2b8c16f5a7d2030"} Nov 03 04:15:01.903578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.903555 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerStarted","Data":"5c07f09e2e5321554c2c98c15c6f0033bd782f73c2e8a792bf7cb2748c6150b5"} Nov 03 04:15:01.903675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.903602 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerStarted","Data":"b5b9f4ade6cf45079610c966f15925d05add6e0ceb3892c019bbf3592339dee6"} Nov 03 04:15:01.999227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:01.999117 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" podStartSLOduration=1.999098843 podStartE2EDuration="1.999098843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:01.998974497 +0000 UTC m=+4393.151662918" watchObservedRunningTime="2026-11-03 04:15:01.999098843 +0000 UTC m=+4393.151787264" Nov 03 04:15:02.120457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.120198 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" podStartSLOduration=2.120179251 podStartE2EDuration="2.120179251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.119339262 +0000 UTC m=+4393.272027685" watchObservedRunningTime="2026-11-03 04:15:02.120179251 +0000 UTC m=+4393.272867673" Nov 03 04:15:02.236347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.236283 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" podStartSLOduration=2.236266804 podStartE2EDuration="2.236266804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.234079797 +0000 UTC m=+4393.386768220" watchObservedRunningTime="2026-11-03 04:15:02.236266804 +0000 UTC m=+4393.388955224" Nov 03 04:15:02.337597 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:02.337556 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:15:02.360812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.360190 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" podStartSLOduration=2.360169064 podStartE2EDuration="2.360169064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.348864531 +0000 UTC m=+4393.501552944" watchObservedRunningTime="2026-11-03 04:15:02.360169064 +0000 UTC m=+4393.512857488" Nov 03 04:15:02.468367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.468301 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" podStartSLOduration=2.468279698 podStartE2EDuration="2.468279698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.466429496 +0000 UTC m=+4393.619117917" watchObservedRunningTime="2026-11-03 04:15:02.468279698 +0000 UTC m=+4393.620968122" Nov 03 04:15:02.584215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.584152 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" podStartSLOduration=2.584132751 podStartE2EDuration="2.584132751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.580350983 +0000 UTC m=+4393.733039405" watchObservedRunningTime="2026-11-03 04:15:02.584132751 +0000 UTC m=+4393.736821173" Nov 03 04:15:02.696585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.696535 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" podStartSLOduration=2.696520129 podStartE2EDuration="2.696520129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.694740996 +0000 UTC m=+4393.847429417" watchObservedRunningTime="2026-11-03 04:15:02.696520129 +0000 UTC m=+4393.849208600" Nov 03 04:15:02.811818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.811761 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" podStartSLOduration=2.811747276 podStartE2EDuration="2.811747276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.809530505 +0000 UTC m=+4393.962218927" watchObservedRunningTime="2026-11-03 04:15:02.811747276 +0000 UTC m=+4393.964435690" Nov 03 04:15:02.912629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.912594 5275 generic.go:356] "Generic (PLEG): container finished" podID="25c01e88-5156-4a40-96bc-727cedb46c81" containerID="f673150a7a1a20e047d9613a520ed3c3eb51ba7942113c1ee0df49d39221a9a9" exitCode=60 Nov 03 04:15:02.912816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.912674 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerDied","Data":"f673150a7a1a20e047d9613a520ed3c3eb51ba7942113c1ee0df49d39221a9a9"} Nov 03 04:15:02.913080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.913060 5275 scope.go:122] "RemoveContainer" containerID="f673150a7a1a20e047d9613a520ed3c3eb51ba7942113c1ee0df49d39221a9a9" Nov 03 04:15:02.914436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.914410 5275 generic.go:356] "Generic (PLEG): container finished" podID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerID="3efce2cbfbc3f88acb4151d7b3f7d09a7f5d962a821a0d993cefafdfd34ec54a" exitCode=60 Nov 03 04:15:02.914538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.914489 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerDied","Data":"3efce2cbfbc3f88acb4151d7b3f7d09a7f5d962a821a0d993cefafdfd34ec54a"} Nov 03 04:15:02.914742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.914726 5275 scope.go:122] "RemoveContainer" containerID="3efce2cbfbc3f88acb4151d7b3f7d09a7f5d962a821a0d993cefafdfd34ec54a" Nov 03 04:15:02.916462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.916441 5275 generic.go:356] "Generic (PLEG): container finished" podID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerID="7f5f951bad78bda5f5d38fe44d587a2a31c809120a52737f5d478380e6361a5c" exitCode=60 Nov 03 04:15:02.916570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.916472 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerDied","Data":"7f5f951bad78bda5f5d38fe44d587a2a31c809120a52737f5d478380e6361a5c"} Nov 03 04:15:02.916718 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.916701 5275 scope.go:122] "RemoveContainer" containerID="7f5f951bad78bda5f5d38fe44d587a2a31c809120a52737f5d478380e6361a5c" Nov 03 04:15:02.919124 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.919105 5275 generic.go:356] "Generic (PLEG): container finished" podID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerID="e7ae328428db2f9f43c438fb3fb6611a8250313acf70693274d781d892d98c7b" exitCode=60 Nov 03 04:15:02.919213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.919166 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerDied","Data":"e7ae328428db2f9f43c438fb3fb6611a8250313acf70693274d781d892d98c7b"} Nov 03 04:15:02.919361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.919346 5275 scope.go:122] "RemoveContainer" containerID="e7ae328428db2f9f43c438fb3fb6611a8250313acf70693274d781d892d98c7b" Nov 03 04:15:02.920778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.920758 5275 generic.go:356] "Generic (PLEG): container finished" podID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerID="517c02293fdce3871cbe61fde0471be30461e29437fc64abdedeae81459ff122" exitCode=60 Nov 03 04:15:02.920886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.920818 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerDied","Data":"517c02293fdce3871cbe61fde0471be30461e29437fc64abdedeae81459ff122"} Nov 03 04:15:02.921096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.921072 5275 scope.go:122] "RemoveContainer" containerID="517c02293fdce3871cbe61fde0471be30461e29437fc64abdedeae81459ff122" Nov 03 04:15:02.922427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.922407 5275 generic.go:356] "Generic (PLEG): container finished" podID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerID="40d2a2963a3cb1b773e754fd589f1a984a0b3f60071356574f43cf1235898c4d" exitCode=60 Nov 03 04:15:02.922541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.922488 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerDied","Data":"40d2a2963a3cb1b773e754fd589f1a984a0b3f60071356574f43cf1235898c4d"} Nov 03 04:15:02.922657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.922642 5275 scope.go:122] "RemoveContainer" containerID="40d2a2963a3cb1b773e754fd589f1a984a0b3f60071356574f43cf1235898c4d" Nov 03 04:15:02.924187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.924167 5275 generic.go:356] "Generic (PLEG): container finished" podID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerID="ad5dc7c072a3ec9b77eb6ef62f608a4f8558af1e7101cba8d1991370e23b6565" exitCode=60 Nov 03 04:15:02.924347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.924191 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerDied","Data":"ad5dc7c072a3ec9b77eb6ef62f608a4f8558af1e7101cba8d1991370e23b6565"} Nov 03 04:15:02.924502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.924485 5275 scope.go:122] "RemoveContainer" containerID="ad5dc7c072a3ec9b77eb6ef62f608a4f8558af1e7101cba8d1991370e23b6565" Nov 03 04:15:02.929133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.929109 5275 generic.go:356] "Generic (PLEG): container finished" podID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerID="3b83bf4cfcc8d5040a8803a6a175a4e2d99d8ac9b81c00554a488a1d833d04a9" exitCode=60 Nov 03 04:15:02.929247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.929193 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerDied","Data":"3b83bf4cfcc8d5040a8803a6a175a4e2d99d8ac9b81c00554a488a1d833d04a9"} Nov 03 04:15:02.929345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.929303 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" podStartSLOduration=2.929285073 podStartE2EDuration="2.929285073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.924958672 +0000 UTC m=+4394.077647094" watchObservedRunningTime="2026-11-03 04:15:02.929285073 +0000 UTC m=+4394.081973495" Nov 03 04:15:02.929659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.929643 5275 scope.go:122] "RemoveContainer" containerID="3b83bf4cfcc8d5040a8803a6a175a4e2d99d8ac9b81c00554a488a1d833d04a9" Nov 03 04:15:02.930966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.930948 5275 generic.go:356] "Generic (PLEG): container finished" podID="f1a81132-06bb-410d-b805-00be6c599442" containerID="eb4f6320628231fb9024e4d4c6b72cd1649246e84c5481d559d71c9ba2c5fb06" exitCode=60 Nov 03 04:15:02.931045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.931023 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerDied","Data":"eb4f6320628231fb9024e4d4c6b72cd1649246e84c5481d559d71c9ba2c5fb06"} Nov 03 04:15:02.931247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.931227 5275 scope.go:122] "RemoveContainer" containerID="eb4f6320628231fb9024e4d4c6b72cd1649246e84c5481d559d71c9ba2c5fb06" Nov 03 04:15:02.932715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.932693 5275 generic.go:356] "Generic (PLEG): container finished" podID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerID="5c07f09e2e5321554c2c98c15c6f0033bd782f73c2e8a792bf7cb2748c6150b5" exitCode=60 Nov 03 04:15:02.932820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.932772 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerDied","Data":"5c07f09e2e5321554c2c98c15c6f0033bd782f73c2e8a792bf7cb2748c6150b5"} Nov 03 04:15:02.932993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.932975 5275 scope.go:122] "RemoveContainer" containerID="5c07f09e2e5321554c2c98c15c6f0033bd782f73c2e8a792bf7cb2748c6150b5" Nov 03 04:15:02.935207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.935137 5275 generic.go:356] "Generic (PLEG): container finished" podID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerID="f9a8c6913fd531e846cf255eb0dbb7803cc4ecfbabe6d6ff3993e841d7934c69" exitCode=60 Nov 03 04:15:02.935417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.935349 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerDied","Data":"f9a8c6913fd531e846cf255eb0dbb7803cc4ecfbabe6d6ff3993e841d7934c69"} Nov 03 04:15:02.935526 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:02.935510 5275 scope.go:122] "RemoveContainer" containerID="f9a8c6913fd531e846cf255eb0dbb7803cc4ecfbabe6d6ff3993e841d7934c69" Nov 03 04:15:03.945428 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.945393 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerStarted","Data":"81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3"} Nov 03 04:15:03.947564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.947529 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerStarted","Data":"ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c"} Nov 03 04:15:03.957891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.951717 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerStarted","Data":"d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67"} Nov 03 04:15:03.957891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.955234 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerStarted","Data":"2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f"} Nov 03 04:15:03.958735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.958386 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerStarted","Data":"1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5"} Nov 03 04:15:03.963887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.961829 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerStarted","Data":"aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d"} Nov 03 04:15:03.965585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.964747 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerStarted","Data":"6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e"} Nov 03 04:15:03.967202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.967173 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerStarted","Data":"e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478"} Nov 03 04:15:03.971452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.971422 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerStarted","Data":"6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea"} Nov 03 04:15:03.974737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.974712 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerStarted","Data":"355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3"} Nov 03 04:15:03.977544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:03.977518 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerStarted","Data":"bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354"} Nov 03 04:15:04.898350 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.898290 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" podStartSLOduration=4.898275727 podStartE2EDuration="4.898275727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:04.897228348 +0000 UTC m=+4396.049916770" watchObservedRunningTime="2026-11-03 04:15:04.898275727 +0000 UTC m=+4396.050964144" Nov 03 04:15:04.989955 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.989918 5275 generic.go:356] "Generic (PLEG): container finished" podID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerID="6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e" exitCode=60 Nov 03 04:15:04.990439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.990004 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerDied","Data":"6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e"} Nov 03 04:15:04.990439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.990064 5275 scope.go:122] "RemoveContainer" containerID="f9a8c6913fd531e846cf255eb0dbb7803cc4ecfbabe6d6ff3993e841d7934c69" Nov 03 04:15:04.990439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.990333 5275 scope.go:122] "RemoveContainer" containerID="6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e" Nov 03 04:15:04.990755 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:04.990712 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5_odh-ai-gateway-infra(95fd1bf8-d2a5-4151-ad29-8cf28dcf5000)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" Nov 03 04:15:04.992351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.992331 5275 generic.go:356] "Generic (PLEG): container finished" podID="25c01e88-5156-4a40-96bc-727cedb46c81" containerID="e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478" exitCode=60 Nov 03 04:15:04.992465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.992399 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerDied","Data":"e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478"} Nov 03 04:15:04.992748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.992711 5275 scope.go:122] "RemoveContainer" containerID="e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478" Nov 03 04:15:04.993081 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:04.993056 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx_odh-ai-gateway-infra(25c01e88-5156-4a40-96bc-727cedb46c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" Nov 03 04:15:04.994711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.994690 5275 generic.go:356] "Generic (PLEG): container finished" podID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerID="6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea" exitCode=60 Nov 03 04:15:04.994818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.994771 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerDied","Data":"6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea"} Nov 03 04:15:04.995111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.995090 5275 scope.go:122] "RemoveContainer" containerID="6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea" Nov 03 04:15:04.995405 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:04.995378 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w_odh-ai-gateway-infra(f644c466-1b9c-495d-b9d8-193c22509b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" Nov 03 04:15:04.997017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.996994 5275 generic.go:356] "Generic (PLEG): container finished" podID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerID="355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3" exitCode=60 Nov 03 04:15:04.997119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.997079 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerDied","Data":"355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3"} Nov 03 04:15:04.997282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.997268 5275 scope.go:122] "RemoveContainer" containerID="355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3" Nov 03 04:15:04.997516 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:04.997493 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l_odh-ai-gateway-infra(4be1a790-6445-4455-875b-3a2417ecc3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" Nov 03 04:15:04.999154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.999133 5275 generic.go:356] "Generic (PLEG): container finished" podID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerID="bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354" exitCode=60 Nov 03 04:15:04.999246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.999155 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerDied","Data":"bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354"} Nov 03 04:15:04.999418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:04.999403 5275 scope.go:122] "RemoveContainer" containerID="bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354" Nov 03 04:15:04.999653 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:04.999630 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc_odh-ai-gateway-infra(7f63c44a-1883-4b56-b061-4ec2128a1ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" Nov 03 04:15:05.001017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.001000 5275 generic.go:356] "Generic (PLEG): container finished" podID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerID="81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3" exitCode=60 Nov 03 04:15:05.001123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.001023 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerDied","Data":"81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3"} Nov 03 04:15:05.001291 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.001273 5275 scope.go:122] "RemoveContainer" containerID="81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3" Nov 03 04:15:05.001530 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.001510 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56_odh-ai-gateway-infra(d1452961-d9b1-4411-ad3b-74b96983de5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" Nov 03 04:15:05.002940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.002921 5275 generic.go:356] "Generic (PLEG): container finished" podID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerID="ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c" exitCode=60 Nov 03 04:15:05.003029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.002992 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerDied","Data":"ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c"} Nov 03 04:15:05.003211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.003197 5275 scope.go:122] "RemoveContainer" containerID="ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c" Nov 03 04:15:05.003418 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.003397 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894655-5szgz_odh-ai-gateway-infra(0c8b3b08-1613-46ad-b053-2634b4d7a396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" Nov 03 04:15:05.004942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.004924 5275 generic.go:356] "Generic (PLEG): container finished" podID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerID="d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67" exitCode=60 Nov 03 04:15:05.005022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.004945 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerDied","Data":"d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67"} Nov 03 04:15:05.005181 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.005166 5275 scope.go:122] "RemoveContainer" containerID="d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67" Nov 03 04:15:05.005373 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.005350 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b_odh-ai-gateway-infra(aa9e270c-cfea-48a8-a370-72ad9d72884e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" Nov 03 04:15:05.005436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.005422 5275 scope.go:122] "RemoveContainer" containerID="f673150a7a1a20e047d9613a520ed3c3eb51ba7942113c1ee0df49d39221a9a9" Nov 03 04:15:05.006968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.006946 5275 generic.go:356] "Generic (PLEG): container finished" podID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerID="2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f" exitCode=60 Nov 03 04:15:05.007064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.007012 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerDied","Data":"2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f"} Nov 03 04:15:05.007206 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.007193 5275 scope.go:122] "RemoveContainer" containerID="2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f" Nov 03 04:15:05.007407 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.007385 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms_odh-ai-gateway-infra(08c94ec6-28aa-4898-8b15-3d13e20b05e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" Nov 03 04:15:05.008748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.008727 5275 generic.go:356] "Generic (PLEG): container finished" podID="f1a81132-06bb-410d-b805-00be6c599442" containerID="1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5" exitCode=60 Nov 03 04:15:05.008825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.008809 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerDied","Data":"1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5"} Nov 03 04:15:05.009027 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.009013 5275 scope.go:122] "RemoveContainer" containerID="1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5" Nov 03 04:15:05.009237 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.009214 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg_odh-ai-gateway-infra(f1a81132-06bb-410d-b805-00be6c599442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" podUID="f1a81132-06bb-410d-b805-00be6c599442" Nov 03 04:15:05.010769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.010749 5275 generic.go:356] "Generic (PLEG): container finished" podID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerID="aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d" exitCode=60 Nov 03 04:15:05.010885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.010775 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerDied","Data":"aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d"} Nov 03 04:15:05.010987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.010973 5275 scope.go:122] "RemoveContainer" containerID="aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d" Nov 03 04:15:05.011162 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:05.011144 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz_odh-ai-gateway-infra(f2c75986-6ac3-456f-b746-6fc49e78f9d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" Nov 03 04:15:05.020028 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.020009 5275 scope.go:122] "RemoveContainer" containerID="3efce2cbfbc3f88acb4151d7b3f7d09a7f5d962a821a0d993cefafdfd34ec54a" Nov 03 04:15:05.032594 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.032569 5275 scope.go:122] "RemoveContainer" containerID="7f5f951bad78bda5f5d38fe44d587a2a31c809120a52737f5d478380e6361a5c" Nov 03 04:15:05.044830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.044811 5275 scope.go:122] "RemoveContainer" containerID="e7ae328428db2f9f43c438fb3fb6611a8250313acf70693274d781d892d98c7b" Nov 03 04:15:05.058638 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.058617 5275 scope.go:122] "RemoveContainer" containerID="517c02293fdce3871cbe61fde0471be30461e29437fc64abdedeae81459ff122" Nov 03 04:15:05.076511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.076491 5275 scope.go:122] "RemoveContainer" containerID="40d2a2963a3cb1b773e754fd589f1a984a0b3f60071356574f43cf1235898c4d" Nov 03 04:15:05.088360 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.088337 5275 scope.go:122] "RemoveContainer" containerID="ad5dc7c072a3ec9b77eb6ef62f608a4f8558af1e7101cba8d1991370e23b6565" Nov 03 04:15:05.100719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.100700 5275 scope.go:122] "RemoveContainer" containerID="3b83bf4cfcc8d5040a8803a6a175a4e2d99d8ac9b81c00554a488a1d833d04a9" Nov 03 04:15:05.113063 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.113044 5275 scope.go:122] "RemoveContainer" containerID="eb4f6320628231fb9024e4d4c6b72cd1649246e84c5481d559d71c9ba2c5fb06" Nov 03 04:15:05.124785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.124765 5275 scope.go:122] "RemoveContainer" containerID="5c07f09e2e5321554c2c98c15c6f0033bd782f73c2e8a792bf7cb2748c6150b5" Nov 03 04:15:05.477236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.477184 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" podStartSLOduration=5.477168039 podStartE2EDuration="5.477168039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:05.472685617 +0000 UTC m=+4396.625374061" watchObservedRunningTime="2026-11-03 04:15:05.477168039 +0000 UTC m=+4396.629856461" Nov 03 04:15:05.836119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.836090 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:15:05.836294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:05.836161 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:15:06.031900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.031864 5275 scope.go:122] "RemoveContainer" containerID="e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478" Nov 03 04:15:06.032325 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.032224 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx_odh-ai-gateway-infra(25c01e88-5156-4a40-96bc-727cedb46c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" Nov 03 04:15:06.033534 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.033511 5275 scope.go:122] "RemoveContainer" containerID="6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea" Nov 03 04:15:06.033801 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.033782 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w_odh-ai-gateway-infra(f644c466-1b9c-495d-b9d8-193c22509b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" Nov 03 04:15:06.035042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.035020 5275 scope.go:122] "RemoveContainer" containerID="355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3" Nov 03 04:15:06.035270 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.035250 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l_odh-ai-gateway-infra(4be1a790-6445-4455-875b-3a2417ecc3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" Nov 03 04:15:06.036425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.036405 5275 scope.go:122] "RemoveContainer" containerID="bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354" Nov 03 04:15:06.036628 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.036604 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc_odh-ai-gateway-infra(7f63c44a-1883-4b56-b061-4ec2128a1ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" Nov 03 04:15:06.037991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.037971 5275 scope.go:122] "RemoveContainer" containerID="81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3" Nov 03 04:15:06.038188 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.038170 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56_odh-ai-gateway-infra(d1452961-d9b1-4411-ad3b-74b96983de5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" Nov 03 04:15:06.039540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.039521 5275 scope.go:122] "RemoveContainer" containerID="ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c" Nov 03 04:15:06.039750 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.039724 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894655-5szgz_odh-ai-gateway-infra(0c8b3b08-1613-46ad-b053-2634b4d7a396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" Nov 03 04:15:06.041037 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.041021 5275 scope.go:122] "RemoveContainer" containerID="d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67" Nov 03 04:15:06.041219 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.041203 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b_odh-ai-gateway-infra(aa9e270c-cfea-48a8-a370-72ad9d72884e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" Nov 03 04:15:06.042439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.042424 5275 scope.go:122] "RemoveContainer" containerID="2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f" Nov 03 04:15:06.042640 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.042619 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms_odh-ai-gateway-infra(08c94ec6-28aa-4898-8b15-3d13e20b05e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" Nov 03 04:15:06.044052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.044033 5275 scope.go:122] "RemoveContainer" containerID="1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5" Nov 03 04:15:06.044225 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.044205 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg_odh-ai-gateway-infra(f1a81132-06bb-410d-b805-00be6c599442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" podUID="f1a81132-06bb-410d-b805-00be6c599442" Nov 03 04:15:06.045474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.045459 5275 scope.go:122] "RemoveContainer" containerID="aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d" Nov 03 04:15:06.045658 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.045636 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz_odh-ai-gateway-infra(f2c75986-6ac3-456f-b746-6fc49e78f9d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" Nov 03 04:15:06.047501 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:06.047483 5275 scope.go:122] "RemoveContainer" containerID="6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e" Nov 03 04:15:06.047699 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:06.047678 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5_odh-ai-gateway-infra(95fd1bf8-d2a5-4151-ad29-8cf28dcf5000)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" Nov 03 04:15:14.977648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:14.977609 5275 scope.go:122] "RemoveContainer" containerID="d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67" Nov 03 04:15:14.978189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:14.977997 5275 scope.go:122] "RemoveContainer" containerID="aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d" Nov 03 04:15:14.978189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:14.978111 5275 scope.go:122] "RemoveContainer" containerID="6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e" Nov 03 04:15:14.978325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:14.978256 5275 scope.go:122] "RemoveContainer" containerID="2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f" Nov 03 04:15:15.978104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:15.978073 5275 scope.go:122] "RemoveContainer" containerID="ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c" Nov 03 04:15:15.978575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:15.978295 5275 scope.go:122] "RemoveContainer" containerID="1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5" Nov 03 04:15:15.978575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:15.978411 5275 scope.go:122] "RemoveContainer" containerID="6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea" Nov 03 04:15:15.978805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:15.978786 5275 scope.go:122] "RemoveContainer" containerID="355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3" Nov 03 04:15:15.978929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:15.978875 5275 scope.go:122] "RemoveContainer" containerID="81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3" Nov 03 04:15:16.158285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.158255 5275 generic.go:356] "Generic (PLEG): container finished" podID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerID="7f53be05821e994a238d0af98c8efb13d462206cf299ff62176541975a1e9e1f" exitCode=60 Nov 03 04:15:16.158511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.158326 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerDied","Data":"7f53be05821e994a238d0af98c8efb13d462206cf299ff62176541975a1e9e1f"} Nov 03 04:15:16.158511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.158393 5275 scope.go:122] "RemoveContainer" containerID="d11a4ae762a71b7909db386b19a9be4ea03374f6c540262c8cf3b6250e1c8d67" Nov 03 04:15:16.158900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.158835 5275 scope.go:122] "RemoveContainer" containerID="7f53be05821e994a238d0af98c8efb13d462206cf299ff62176541975a1e9e1f" Nov 03 04:15:16.159323 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:16.159283 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b_odh-ai-gateway-infra(aa9e270c-cfea-48a8-a370-72ad9d72884e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" Nov 03 04:15:16.160893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.160867 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerStarted","Data":"caa17262b49b1104dd028e9c75aa909ea5de3635f993d48fc70d3ef27f03354a"} Nov 03 04:15:16.163815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.163787 5275 generic.go:356] "Generic (PLEG): container finished" podID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerID="dd6a0ddb2c89868845cca71e69fc87f10854558426e0b822357796f22da3f7d1" exitCode=60 Nov 03 04:15:16.163931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.163868 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerDied","Data":"dd6a0ddb2c89868845cca71e69fc87f10854558426e0b822357796f22da3f7d1"} Nov 03 04:15:16.164264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.164245 5275 scope.go:122] "RemoveContainer" containerID="dd6a0ddb2c89868845cca71e69fc87f10854558426e0b822357796f22da3f7d1" Nov 03 04:15:16.164598 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:16.164575 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz_odh-ai-gateway-infra(f2c75986-6ac3-456f-b746-6fc49e78f9d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" Nov 03 04:15:16.166393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.166369 5275 generic.go:356] "Generic (PLEG): container finished" podID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerID="fabecec07396867cd1a7c43e523950479053c68cb32fc4c19c36bfe5da796c11" exitCode=60 Nov 03 04:15:16.166529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.166444 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerDied","Data":"fabecec07396867cd1a7c43e523950479053c68cb32fc4c19c36bfe5da796c11"} Nov 03 04:15:16.166824 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.166809 5275 scope.go:122] "RemoveContainer" containerID="fabecec07396867cd1a7c43e523950479053c68cb32fc4c19c36bfe5da796c11" Nov 03 04:15:16.167195 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:16.167172 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5_odh-ai-gateway-infra(95fd1bf8-d2a5-4151-ad29-8cf28dcf5000)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" Nov 03 04:15:16.174286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.174258 5275 scope.go:122] "RemoveContainer" containerID="aa350cea004fa6342695a4d3376908c657545021d893c7e6fb5798de5c766a9d" Nov 03 04:15:16.195543 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.195522 5275 scope.go:122] "RemoveContainer" containerID="6aad2bc49646ff05161a16d8587a3d2b83c90e40f1534dfc4ba22311ebc8d01e" Nov 03 04:15:16.978065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.978035 5275 scope.go:122] "RemoveContainer" containerID="e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478" Nov 03 04:15:16.978278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:16.978193 5275 scope.go:122] "RemoveContainer" containerID="bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354" Nov 03 04:15:17.179101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.179065 5275 generic.go:356] "Generic (PLEG): container finished" podID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerID="08fb34408d3e26349347b4650b9dbe81787a4254fc4a7fc9d8ec52909a081eee" exitCode=60 Nov 03 04:15:17.179101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.179089 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerDied","Data":"08fb34408d3e26349347b4650b9dbe81787a4254fc4a7fc9d8ec52909a081eee"} Nov 03 04:15:17.179383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.179171 5275 scope.go:122] "RemoveContainer" containerID="6cd9d98e91335349355bc16d76679a0488ad692342c3f12cc3db4d07b14e05ea" Nov 03 04:15:17.179632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.179614 5275 scope.go:122] "RemoveContainer" containerID="08fb34408d3e26349347b4650b9dbe81787a4254fc4a7fc9d8ec52909a081eee" Nov 03 04:15:17.180059 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:17.180030 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w_odh-ai-gateway-infra(f644c466-1b9c-495d-b9d8-193c22509b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" Nov 03 04:15:17.181470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.181445 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerStarted","Data":"2fcf6fa810d67249b4d0baf779f7f1ba92732a2540223067b8bf089bc731cec1"} Nov 03 04:15:17.183763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.183730 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerStarted","Data":"40e47894696108dded880c7803e433fc02ecc9ce4e6749b070574a73b9ec31a7"} Nov 03 04:15:17.186356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.186322 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerStarted","Data":"22815c1dae3432720c417284b6c127a38eaf0ca9a9e81fdbe40d1d6a6f539bd9"} Nov 03 04:15:17.192041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.190991 5275 generic.go:356] "Generic (PLEG): container finished" podID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerID="caa17262b49b1104dd028e9c75aa909ea5de3635f993d48fc70d3ef27f03354a" exitCode=60 Nov 03 04:15:17.192041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.191070 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerDied","Data":"caa17262b49b1104dd028e9c75aa909ea5de3635f993d48fc70d3ef27f03354a"} Nov 03 04:15:17.192041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.191818 5275 scope.go:122] "RemoveContainer" containerID="caa17262b49b1104dd028e9c75aa909ea5de3635f993d48fc70d3ef27f03354a" Nov 03 04:15:17.192328 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:17.192286 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms_odh-ai-gateway-infra(08c94ec6-28aa-4898-8b15-3d13e20b05e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" Nov 03 04:15:17.193578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.193554 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerStarted","Data":"3c6a916b7a20e55998751b25d710cf72729066aced91ad5483153fd6bd60bdbe"} Nov 03 04:15:17.207984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.207962 5275 scope.go:122] "RemoveContainer" containerID="2203b7f7c0c95316f56d18a15d62606012d8cd4a5c36db2be045946843d27e1f" Nov 03 04:15:17.286664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.286588 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:17.400532 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.400504 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:17.518357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.518322 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:17.548798 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.548727 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:17.579436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.579407 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:17.634375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.634342 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:17.675037 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.675005 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\") pod \"aa9e270c-cfea-48a8-a370-72ad9d72884e\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " Nov 03 04:15:17.675270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.675256 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\") pod \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " Nov 03 04:15:17.675349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.675339 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\" (UniqueName: \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\") pod \"aa9e270c-cfea-48a8-a370-72ad9d72884e\" (UID: \"aa9e270c-cfea-48a8-a370-72ad9d72884e\") " Nov 03 04:15:17.675532 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.675522 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\" (UniqueName: \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\") pod \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\" (UID: \"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000\") " Nov 03 04:15:17.677425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.677391 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca" pod "95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" (UID: "95fd1bf8-d2a5-4151-ad29-8cf28dcf5000"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:17.677624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.677410 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca" pod "aa9e270c-cfea-48a8-a370-72ad9d72884e" (UID: "aa9e270c-cfea-48a8-a370-72ad9d72884e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:17.678815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.678790 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg" pod "95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" (UID: "95fd1bf8-d2a5-4151-ad29-8cf28dcf5000"). InnerVolumeSpecName "kube-api-access-fl7rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:17.680524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.680496 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw" pod "aa9e270c-cfea-48a8-a370-72ad9d72884e" (UID: "aa9e270c-cfea-48a8-a370-72ad9d72884e"). InnerVolumeSpecName "kube-api-access-jrkqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:17.694335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.694310 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:17.776776 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.776738 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\") pod \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " Nov 03 04:15:17.776972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.776893 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\" (UniqueName: \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\") pod \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\" (UID: \"f2c75986-6ac3-456f-b746-6fc49e78f9d5\") " Nov 03 04:15:17.777126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777104 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca" pod "f2c75986-6ac3-456f-b746-6fc49e78f9d5" (UID: "f2c75986-6ac3-456f-b746-6fc49e78f9d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:17.777453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777434 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9e270c-cfea-48a8-a370-72ad9d72884e-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:17.777505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777461 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:17.777505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777477 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrkqw\" (UniqueName: \"kubernetes.io/projected/aa9e270c-cfea-48a8-a370-72ad9d72884e-kube-api-access-jrkqw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:17.777505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777489 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c75986-6ac3-456f-b746-6fc49e78f9d5-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:17.777505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.777498 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7rg\" (UniqueName: \"kubernetes.io/projected/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000-kube-api-access-fl7rg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:17.779265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.779248 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg" pod "f2c75986-6ac3-456f-b746-6fc49e78f9d5" (UID: "f2c75986-6ac3-456f-b746-6fc49e78f9d5"). InnerVolumeSpecName "kube-api-access-nm7bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:17.878910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:17.878816 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7bg\" (UniqueName: \"kubernetes.io/projected/f2c75986-6ac3-456f-b746-6fc49e78f9d5-kube-api-access-nm7bg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:18.207087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.207055 5275 generic.go:356] "Generic (PLEG): container finished" podID="25c01e88-5156-4a40-96bc-727cedb46c81" containerID="2ac36ae4dfc181de20ec59254312c3f5cc44e0000ccc7ef09aaa869bb44c7691" exitCode=60 Nov 03 04:15:18.207564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.207129 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerDied","Data":"2ac36ae4dfc181de20ec59254312c3f5cc44e0000ccc7ef09aaa869bb44c7691"} Nov 03 04:15:18.207564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.207202 5275 scope.go:122] "RemoveContainer" containerID="e4f3656f5b8e15c7a2af21aef4f698b2906162ced213493c1508e4693bb74478" Nov 03 04:15:18.207704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.207620 5275 scope.go:122] "RemoveContainer" containerID="2ac36ae4dfc181de20ec59254312c3f5cc44e0000ccc7ef09aaa869bb44c7691" Nov 03 04:15:18.208027 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.208005 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx_odh-ai-gateway-infra(25c01e88-5156-4a40-96bc-727cedb46c81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" Nov 03 04:15:18.211062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.211008 5275 generic.go:356] "Generic (PLEG): container finished" podID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerID="2fcf6fa810d67249b4d0baf779f7f1ba92732a2540223067b8bf089bc731cec1" exitCode=60 Nov 03 04:15:18.211240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.211077 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerDied","Data":"2fcf6fa810d67249b4d0baf779f7f1ba92732a2540223067b8bf089bc731cec1"} Nov 03 04:15:18.211610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.211589 5275 scope.go:122] "RemoveContainer" containerID="2fcf6fa810d67249b4d0baf779f7f1ba92732a2540223067b8bf089bc731cec1" Nov 03 04:15:18.212034 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.211997 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l_odh-ai-gateway-infra(4be1a790-6445-4455-875b-3a2417ecc3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" Nov 03 04:15:18.213472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.213450 5275 generic.go:356] "Generic (PLEG): container finished" podID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerID="f76af66f916788c24ced8f1ddb005a3f0a3bb09f7a6502fc0c0f09e690721539" exitCode=60 Nov 03 04:15:18.213590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.213516 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerDied","Data":"f76af66f916788c24ced8f1ddb005a3f0a3bb09f7a6502fc0c0f09e690721539"} Nov 03 04:15:18.214043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.214018 5275 scope.go:122] "RemoveContainer" containerID="f76af66f916788c24ced8f1ddb005a3f0a3bb09f7a6502fc0c0f09e690721539" Nov 03 04:15:18.214389 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.214366 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc_odh-ai-gateway-infra(7f63c44a-1883-4b56-b061-4ec2128a1ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" Nov 03 04:15:18.216561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.216543 5275 generic.go:356] "Generic (PLEG): container finished" podID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerID="40e47894696108dded880c7803e433fc02ecc9ce4e6749b070574a73b9ec31a7" exitCode=60 Nov 03 04:15:18.216664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.216615 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerDied","Data":"40e47894696108dded880c7803e433fc02ecc9ce4e6749b070574a73b9ec31a7"} Nov 03 04:15:18.217063 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.217037 5275 scope.go:122] "RemoveContainer" containerID="40e47894696108dded880c7803e433fc02ecc9ce4e6749b070574a73b9ec31a7" Nov 03 04:15:18.217378 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.217356 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56_odh-ai-gateway-infra(d1452961-d9b1-4411-ad3b-74b96983de5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" Nov 03 04:15:18.218422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.218404 5275 generic.go:356] "Generic (PLEG): container finished" podID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerID="22815c1dae3432720c417284b6c127a38eaf0ca9a9e81fdbe40d1d6a6f539bd9" exitCode=60 Nov 03 04:15:18.218505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.218478 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerDied","Data":"22815c1dae3432720c417284b6c127a38eaf0ca9a9e81fdbe40d1d6a6f539bd9"} Nov 03 04:15:18.218835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.218799 5275 scope.go:122] "RemoveContainer" containerID="22815c1dae3432720c417284b6c127a38eaf0ca9a9e81fdbe40d1d6a6f539bd9" Nov 03 04:15:18.219124 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.219100 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894655-5szgz_odh-ai-gateway-infra(0c8b3b08-1613-46ad-b053-2634b4d7a396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" Nov 03 04:15:18.220080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.220063 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" Nov 03 04:15:18.220195 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.220091 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b" event={"ID":"aa9e270c-cfea-48a8-a370-72ad9d72884e","Type":"ContainerDied","Data":"c197fcf403f86df83b9244ec541c42c2db25b79cc7ab177223d09a0f579dae19"} Nov 03 04:15:18.227708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.225342 5275 generic.go:356] "Generic (PLEG): container finished" podID="f1a81132-06bb-410d-b805-00be6c599442" containerID="3c6a916b7a20e55998751b25d710cf72729066aced91ad5483153fd6bd60bdbe" exitCode=60 Nov 03 04:15:18.227708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.225424 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerDied","Data":"3c6a916b7a20e55998751b25d710cf72729066aced91ad5483153fd6bd60bdbe"} Nov 03 04:15:18.227708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.225717 5275 scope.go:122] "RemoveContainer" containerID="3c6a916b7a20e55998751b25d710cf72729066aced91ad5483153fd6bd60bdbe" Nov 03 04:15:18.227708 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:18.226380 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg_odh-ai-gateway-infra(f1a81132-06bb-410d-b805-00be6c599442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" podUID="f1a81132-06bb-410d-b805-00be6c599442" Nov 03 04:15:18.228969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.228940 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" event={"ID":"f2c75986-6ac3-456f-b746-6fc49e78f9d5","Type":"ContainerDied","Data":"b5b9f4ade6cf45079610c966f15925d05add6e0ceb3892c019bbf3592339dee6"} Nov 03 04:15:18.229103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.229040 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz" Nov 03 04:15:18.237032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.236995 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" event={"ID":"95fd1bf8-d2a5-4151-ad29-8cf28dcf5000","Type":"ContainerDied","Data":"e09a747df14b7a512ee1fb056111b5067f0dffec2a8b7fc716a2483f99c43226"} Nov 03 04:15:18.237032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.237018 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5" Nov 03 04:15:18.286262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.286223 5275 scope.go:122] "RemoveContainer" containerID="355d46eb4ecc66bb24ff75040e1d5bf5226eb8ddece60fc4dc747de63ea9f6f3" Nov 03 04:15:18.305089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.302425 5275 scope.go:122] "RemoveContainer" containerID="bdb5a25a6469a086cdd05f95599bca9be9d99974462cac98259e91ebfc2e6354" Nov 03 04:15:18.312642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.312613 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:18.348985 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.348956 5275 scope.go:122] "RemoveContainer" containerID="81a15f25cc6629dc6745fe99c2502107d48614051ae0ba690b132afd96b514f3" Nov 03 04:15:18.377680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.377515 5275 scope.go:122] "RemoveContainer" containerID="ed28c199498c8b5088dbbccf245a7d821339f3bfb11cb51d76944a73a2a3b39c" Nov 03 04:15:18.422888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.420041 5275 scope.go:122] "RemoveContainer" containerID="7f53be05821e994a238d0af98c8efb13d462206cf299ff62176541975a1e9e1f" Nov 03 04:15:18.427213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.427132 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:18.438448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.438355 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:18.446129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.446109 5275 scope.go:122] "RemoveContainer" containerID="1fddacf81b88f8316352a6c3aea80b9fc8bb46021783f2686463802f7d3580b5" Nov 03 04:15:18.468560 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.468536 5275 scope.go:122] "RemoveContainer" containerID="dd6a0ddb2c89868845cca71e69fc87f10854558426e0b822357796f22da3f7d1" Nov 03 04:15:18.479731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.479711 5275 scope.go:122] "RemoveContainer" containerID="fabecec07396867cd1a7c43e523950479053c68cb32fc4c19c36bfe5da796c11" Nov 03 04:15:18.490132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.490107 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:18.546641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.546596 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:18.566550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.566519 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:18.574648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.574619 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-286a8c-29894655-k5x9b"] Nov 03 04:15:18.588112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588078 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\") pod \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " Nov 03 04:15:18.588289 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588164 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\" (UniqueName: \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\") pod \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\" (UID: \"08c94ec6-28aa-4898-8b15-3d13e20b05e2\") " Nov 03 04:15:18.588289 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588208 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\") pod \"f644c466-1b9c-495d-b9d8-193c22509b2e\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " Nov 03 04:15:18.588406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588390 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\" (UniqueName: \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\") pod \"f644c466-1b9c-495d-b9d8-193c22509b2e\" (UID: \"f644c466-1b9c-495d-b9d8-193c22509b2e\") " Nov 03 04:15:18.588600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588571 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca" pod "08c94ec6-28aa-4898-8b15-3d13e20b05e2" (UID: "08c94ec6-28aa-4898-8b15-3d13e20b05e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:18.588687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588644 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca" pod "f644c466-1b9c-495d-b9d8-193c22509b2e" (UID: "f644c466-1b9c-495d-b9d8-193c22509b2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:18.588960 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588942 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c94ec6-28aa-4898-8b15-3d13e20b05e2-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:18.589025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.588963 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f644c466-1b9c-495d-b9d8-193c22509b2e-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:18.590535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.590515 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs" pod "08c94ec6-28aa-4898-8b15-3d13e20b05e2" (UID: "08c94ec6-28aa-4898-8b15-3d13e20b05e2"). InnerVolumeSpecName "kube-api-access-l85bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:18.590582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.590565 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng" pod "f644c466-1b9c-495d-b9d8-193c22509b2e" (UID: "f644c466-1b9c-495d-b9d8-193c22509b2e"). InnerVolumeSpecName "kube-api-access-2ldng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:18.690629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.690535 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldng\" (UniqueName: \"kubernetes.io/projected/f644c466-1b9c-495d-b9d8-193c22509b2e-kube-api-access-2ldng\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:18.690629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.690568 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85bs\" (UniqueName: \"kubernetes.io/projected/08c94ec6-28aa-4898-8b15-3d13e20b05e2-kube-api-access-l85bs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:18.774417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.774384 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:18.797639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.797601 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:18.808440 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.808401 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0aa6b9-29894655-fhtg5"] Nov 03 04:15:18.986936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.986832 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" path="/var/lib/kubelet/pods/95fd1bf8-d2a5-4151-ad29-8cf28dcf5000/volumes" Nov 03 04:15:18.987432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:18.987411 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" path="/var/lib/kubelet/pods/aa9e270c-cfea-48a8-a370-72ad9d72884e/volumes" Nov 03 04:15:19.003691 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.003664 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:19.147322 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.147284 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:19.155356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.155327 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-10120d-29894655-mccmz"] Nov 03 04:15:19.248900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.248787 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" Nov 03 04:15:19.248900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.248794 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w" event={"ID":"f644c466-1b9c-495d-b9d8-193c22509b2e","Type":"ContainerDied","Data":"5a6161151721b65f9904ece7675b096dad235f792e4094951470954f928c4f4a"} Nov 03 04:15:19.249431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.248951 5275 scope.go:122] "RemoveContainer" containerID="08fb34408d3e26349347b4650b9dbe81787a4254fc4a7fc9d8ec52909a081eee" Nov 03 04:15:19.259922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.259895 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" Nov 03 04:15:19.260370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.259895 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms" event={"ID":"08c94ec6-28aa-4898-8b15-3d13e20b05e2","Type":"ContainerDied","Data":"d59193714d21ad5e6234288d8e1d07c49eb46dc4cae43437eadab6e76bbe3662"} Nov 03 04:15:19.264941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.264919 5275 scope.go:122] "RemoveContainer" containerID="caa17262b49b1104dd028e9c75aa909ea5de3635f993d48fc70d3ef27f03354a" Nov 03 04:15:19.535890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.535284 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:19.541277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.541250 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:19.556597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.556563 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:19.560977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.560953 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:19.603539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.603506 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\" (UniqueName: \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\") pod \"d1452961-d9b1-4411-ad3b-74b96983de5f\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " Nov 03 04:15:19.603702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.603560 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\") pod \"d1452961-d9b1-4411-ad3b-74b96983de5f\" (UID: \"d1452961-d9b1-4411-ad3b-74b96983de5f\") " Nov 03 04:15:19.603785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.603695 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\" (UniqueName: \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\") pod \"4be1a790-6445-4455-875b-3a2417ecc3c7\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " Nov 03 04:15:19.603785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.603769 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\") pod \"4be1a790-6445-4455-875b-3a2417ecc3c7\" (UID: \"4be1a790-6445-4455-875b-3a2417ecc3c7\") " Nov 03 04:15:19.603986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.603964 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca" pod "d1452961-d9b1-4411-ad3b-74b96983de5f" (UID: "d1452961-d9b1-4411-ad3b-74b96983de5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:19.604140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.604118 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca" pod "4be1a790-6445-4455-875b-3a2417ecc3c7" (UID: "4be1a790-6445-4455-875b-3a2417ecc3c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:19.604471 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.604452 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1452961-d9b1-4411-ad3b-74b96983de5f-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.604568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.604477 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1a790-6445-4455-875b-3a2417ecc3c7-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.606279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.606255 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z" pod "4be1a790-6445-4455-875b-3a2417ecc3c7" (UID: "4be1a790-6445-4455-875b-3a2417ecc3c7"). InnerVolumeSpecName "kube-api-access-zb57z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:19.606380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.606330 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7" pod "d1452961-d9b1-4411-ad3b-74b96983de5f" (UID: "d1452961-d9b1-4411-ad3b-74b96983de5f"). InnerVolumeSpecName "kube-api-access-xscv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:19.705800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.705762 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\" (UniqueName: \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\") pod \"0c8b3b08-1613-46ad-b053-2634b4d7a396\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " Nov 03 04:15:19.706045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.705984 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\" (UniqueName: \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\") pod \"f1a81132-06bb-410d-b805-00be6c599442\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " Nov 03 04:15:19.706209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.706078 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\") pod \"0c8b3b08-1613-46ad-b053-2634b4d7a396\" (UID: \"0c8b3b08-1613-46ad-b053-2634b4d7a396\") " Nov 03 04:15:19.706209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.706169 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\") pod \"f1a81132-06bb-410d-b805-00be6c599442\" (UID: \"f1a81132-06bb-410d-b805-00be6c599442\") " Nov 03 04:15:19.706711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.706691 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xscv7\" (UniqueName: \"kubernetes.io/projected/d1452961-d9b1-4411-ad3b-74b96983de5f-kube-api-access-xscv7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.706780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.706716 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb57z\" (UniqueName: \"kubernetes.io/projected/4be1a790-6445-4455-875b-3a2417ecc3c7-kube-api-access-zb57z\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.707110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.707086 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca" pod "f1a81132-06bb-410d-b805-00be6c599442" (UID: "f1a81132-06bb-410d-b805-00be6c599442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:19.707766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.707708 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca" pod "0c8b3b08-1613-46ad-b053-2634b4d7a396" (UID: "0c8b3b08-1613-46ad-b053-2634b4d7a396"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:19.709526 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.709474 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9" pod "f1a81132-06bb-410d-b805-00be6c599442" (UID: "f1a81132-06bb-410d-b805-00be6c599442"). InnerVolumeSpecName "kube-api-access-lc4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:19.710218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.710179 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987" pod "0c8b3b08-1613-46ad-b053-2634b4d7a396" (UID: "0c8b3b08-1613-46ad-b053-2634b4d7a396"). InnerVolumeSpecName "kube-api-access-5j987". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:19.808237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.808200 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4r9\" (UniqueName: \"kubernetes.io/projected/f1a81132-06bb-410d-b805-00be6c599442-kube-api-access-lc4r9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.808237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.808237 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8b3b08-1613-46ad-b053-2634b4d7a396-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.808427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.808252 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a81132-06bb-410d-b805-00be6c599442-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.808427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.808266 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j987\" (UniqueName: \"kubernetes.io/projected/0c8b3b08-1613-46ad-b053-2634b4d7a396-kube-api-access-5j987\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:19.928971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:19.928932 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:20.107611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.107536 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:20.162103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.162017 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:20.166780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.166753 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894655-zwq8w"] Nov 03 04:15:20.214209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.214154 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\") pod \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " Nov 03 04:15:20.214414 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.214310 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\" (UniqueName: \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\") pod \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\" (UID: \"7f63c44a-1883-4b56-b061-4ec2128a1ea2\") " Nov 03 04:15:20.214559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.214529 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca" pod "7f63c44a-1883-4b56-b061-4ec2128a1ea2" (UID: "7f63c44a-1883-4b56-b061-4ec2128a1ea2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:20.214943 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.214919 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f63c44a-1883-4b56-b061-4ec2128a1ea2-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.216756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.216726 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw" pod "7f63c44a-1883-4b56-b061-4ec2128a1ea2" (UID: "7f63c44a-1883-4b56-b061-4ec2128a1ea2"). InnerVolumeSpecName "kube-api-access-r8bsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:20.284308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.284267 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" event={"ID":"4be1a790-6445-4455-875b-3a2417ecc3c7","Type":"ContainerDied","Data":"be89a086242fdf6efcbd70a48bae44107eba11cd34462fc162f574a987093c84"} Nov 03 04:15:20.284757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.284339 5275 scope.go:122] "RemoveContainer" containerID="2fcf6fa810d67249b4d0baf779f7f1ba92732a2540223067b8bf089bc731cec1" Nov 03 04:15:20.284757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.284440 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l" Nov 03 04:15:20.290915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.290883 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" event={"ID":"7f63c44a-1883-4b56-b061-4ec2128a1ea2","Type":"ContainerDied","Data":"e011f5ff931068354f7bcf53f1d9a9938a7fa3cb4a3081a0b98b8380cb9ce9c7"} Nov 03 04:15:20.291044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.291002 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc" Nov 03 04:15:20.314697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.302371 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" event={"ID":"d1452961-d9b1-4411-ad3b-74b96983de5f","Type":"ContainerDied","Data":"ff431d3bce589ce2dbe6a7da120b2f3f411fc5f91bc48e6ed3715f09f93dd52f"} Nov 03 04:15:20.314697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.302464 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56" Nov 03 04:15:20.314697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.304348 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" event={"ID":"0c8b3b08-1613-46ad-b053-2634b4d7a396","Type":"ContainerDied","Data":"b219ee6a4e313131b7af41349fe1b48f15792965d7942568745b349bece64db6"} Nov 03 04:15:20.314697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.304447 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz" Nov 03 04:15:20.315985 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.315955 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bsw\" (UniqueName: \"kubernetes.io/projected/7f63c44a-1883-4b56-b061-4ec2128a1ea2-kube-api-access-r8bsw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.322587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.322115 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" event={"ID":"f1a81132-06bb-410d-b805-00be6c599442","Type":"ContainerDied","Data":"0893cc5c4d4e85debd6908dde0ac3be82e519075f096cc49d2b8c16f5a7d2030"} Nov 03 04:15:20.322587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.322219 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg" Nov 03 04:15:20.334493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.334410 5275 scope.go:122] "RemoveContainer" containerID="f76af66f916788c24ced8f1ddb005a3f0a3bb09f7a6502fc0c0f09e690721539" Nov 03 04:15:20.349832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.349808 5275 scope.go:122] "RemoveContainer" containerID="40e47894696108dded880c7803e433fc02ecc9ce4e6749b070574a73b9ec31a7" Nov 03 04:15:20.360328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.360309 5275 scope.go:122] "RemoveContainer" containerID="22815c1dae3432720c417284b6c127a38eaf0ca9a9e81fdbe40d1d6a6f539bd9" Nov 03 04:15:20.370344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.370307 5275 scope.go:122] "RemoveContainer" containerID="3c6a916b7a20e55998751b25d710cf72729066aced91ad5483153fd6bd60bdbe" Nov 03 04:15:20.394296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.394266 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:20.458438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.458399 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:20.468559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.468523 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ed4ecc-29894655-t9cms"] Nov 03 04:15:20.616525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.616498 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:20.670875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.670382 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:20.674874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.672729 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894655-2nxhc"] Nov 03 04:15:20.742907 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.742839 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\" (UniqueName: \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\") pod \"25c01e88-5156-4a40-96bc-727cedb46c81\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " Nov 03 04:15:20.743065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.742959 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\") pod \"25c01e88-5156-4a40-96bc-727cedb46c81\" (UID: \"25c01e88-5156-4a40-96bc-727cedb46c81\") " Nov 03 04:15:20.743307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.743284 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca" pod "25c01e88-5156-4a40-96bc-727cedb46c81" (UID: "25c01e88-5156-4a40-96bc-727cedb46c81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:20.743425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.743405 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c01e88-5156-4a40-96bc-727cedb46c81-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.745322 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.745300 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr" pod "25c01e88-5156-4a40-96bc-727cedb46c81" (UID: "25c01e88-5156-4a40-96bc-727cedb46c81"). InnerVolumeSpecName "kube-api-access-2vnwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:20.750014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.749988 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:20.757211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.757182 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894655-5szgz"] Nov 03 04:15:20.844801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.844765 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnwr\" (UniqueName: \"kubernetes.io/projected/25c01e88-5156-4a40-96bc-727cedb46c81-kube-api-access-2vnwr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.989792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.989715 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" path="/var/lib/kubelet/pods/08c94ec6-28aa-4898-8b15-3d13e20b05e2/volumes" Nov 03 04:15:20.990248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.990233 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" path="/var/lib/kubelet/pods/0c8b3b08-1613-46ad-b053-2634b4d7a396/volumes" Nov 03 04:15:20.990609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.990596 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" path="/var/lib/kubelet/pods/7f63c44a-1883-4b56-b061-4ec2128a1ea2/volumes" Nov 03 04:15:20.991326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.991312 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" path="/var/lib/kubelet/pods/f2c75986-6ac3-456f-b746-6fc49e78f9d5/volumes" Nov 03 04:15:20.991686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.991674 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" path="/var/lib/kubelet/pods/f644c466-1b9c-495d-b9d8-193c22509b2e/volumes" Nov 03 04:15:20.992210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.992195 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:20.992250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:20.992221 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894655-bm4gg"] Nov 03 04:15:21.098561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.098526 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:21.105218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.105189 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-66d902-29894655-djz4l"] Nov 03 04:15:21.247601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.244120 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:21.268875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.259689 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f3e02b-29894655-kwv56"] Nov 03 04:15:21.333208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.333178 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" Nov 03 04:15:21.333675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.333176 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx" event={"ID":"25c01e88-5156-4a40-96bc-727cedb46c81","Type":"ContainerDied","Data":"cab37b35625ed72e4b1391bccd687c8b6ece683a321c23674bd6249b99d65b42"} Nov 03 04:15:21.333675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.333313 5275 scope.go:122] "RemoveContainer" containerID="2ac36ae4dfc181de20ec59254312c3f5cc44e0000ccc7ef09aaa869bb44c7691" Nov 03 04:15:21.455816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.455779 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:21.461283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:21.461254 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9daccc-29894655-v8hwx"] Nov 03 04:15:22.991929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:22.991827 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" path="/var/lib/kubelet/pods/25c01e88-5156-4a40-96bc-727cedb46c81/volumes" Nov 03 04:15:22.992347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:22.992331 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" path="/var/lib/kubelet/pods/4be1a790-6445-4455-875b-3a2417ecc3c7/volumes" Nov 03 04:15:22.992739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:22.992724 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" path="/var/lib/kubelet/pods/d1452961-d9b1-4411-ad3b-74b96983de5f/volumes" Nov 03 04:15:22.993469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:22.993453 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a81132-06bb-410d-b805-00be6c599442" path="/var/lib/kubelet/pods/f1a81132-06bb-410d-b805-00be6c599442/volumes" Nov 03 04:15:32.182247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:32.182212 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:15:32.182755 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:32.182433 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" containerID="cri-o://70dd152f18d8504996a52031a5f3be42197039d4af416177ed78bf9c39ebf951" gracePeriod=30 Nov 03 04:15:32.361871 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:32.360994 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:15:35.505677 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:35.505635 5275 patch_prober.go:29] interesting pod/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:15:35.506133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:35.505716 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:15:37.471834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.471799 5275 generic.go:356] "Generic (PLEG): container finished" podID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerID="70dd152f18d8504996a52031a5f3be42197039d4af416177ed78bf9c39ebf951" exitCode=0 Nov 03 04:15:37.472237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.471885 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerDied","Data":"70dd152f18d8504996a52031a5f3be42197039d4af416177ed78bf9c39ebf951"} Nov 03 04:15:37.472237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.471974 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" event={"ID":"3dfded98-72c8-4567-b05c-e5a1386f8f1a","Type":"ContainerDied","Data":"ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5"} Nov 03 04:15:37.472237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.472005 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4e89943afe2a569082adcdfff5fd1697c1261c193278c25477d0b46acf3ef5" Nov 03 04:15:37.472237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.472025 5275 scope.go:122] "RemoveContainer" containerID="f5a3da2b1b42a04526626524c0151ff0b60fd694610ac9bb6fedb3c07268203f" Nov 03 04:15:37.484497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.484176 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:15:37.576339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576306 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576376 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576402 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576486 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576524 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576567 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576633 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576661 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:15:37.576862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576689 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.576862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576727 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:15:37.577145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576897 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\") pod \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\" (UID: \"3dfded98-72c8-4567-b05c-e5a1386f8f1a\") " Nov 03 04:15:37.577145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.576908 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:15:37.577145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577025 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:15:37.577314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577235 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:37.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577648 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577674 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577689 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577703 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.577744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.577720 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.580339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.579334 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "kube-api-access-s6s9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:37.580339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.579603 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:15:37.580339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.579644 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:37.580339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.579752 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo" pod "3dfded98-72c8-4567-b05c-e5a1386f8f1a" (UID: "3dfded98-72c8-4567-b05c-e5a1386f8f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:15:37.679604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.679569 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.679604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.679598 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.679604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.679609 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfded98-72c8-4567-b05c-e5a1386f8f1a-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:37.679976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:37.679619 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6s9c\" (UniqueName: \"kubernetes.io/projected/3dfded98-72c8-4567-b05c-e5a1386f8f1a-kube-api-access-s6s9c\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:38.483370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:38.483336 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh" Nov 03 04:15:38.767492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:38.767389 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:15:38.773444 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:38.773401 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ed4ecc-openshift-default-9b84595c9-drdrh"] Nov 03 04:15:38.988323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:38.988289 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" path="/var/lib/kubelet/pods/3dfded98-72c8-4567-b05c-e5a1386f8f1a/volumes" Nov 03 04:15:40.334493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:40.334455 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:15:40.440209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:40.440179 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:15:45.335716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.335668 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" podUID="38fb8aa4-6e7a-4570-904b-579022f834d7" containerName="payload-processing" containerID="cri-o://f043e1704daa3804cccd5a8eb9e04936c7b759ced25844e4217933c26ccfddbb" gracePeriod=25 Nov 03 04:15:45.440905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.440831 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" podUID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" containerName="payload-pre-processing" containerID="cri-o://6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a" gracePeriod=25 Nov 03 04:15:45.546894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.546796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" event={"ID":"38fb8aa4-6e7a-4570-904b-579022f834d7","Type":"ContainerDied","Data":"f043e1704daa3804cccd5a8eb9e04936c7b759ced25844e4217933c26ccfddbb"} Nov 03 04:15:45.547227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.546738 5275 generic.go:356] "Generic (PLEG): container finished" podID="38fb8aa4-6e7a-4570-904b-579022f834d7" containerID="f043e1704daa3804cccd5a8eb9e04936c7b759ced25844e4217933c26ccfddbb" exitCode=0 Nov 03 04:15:45.631005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.630977 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:15:45.721708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.721670 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:15:45.791917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.791882 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\") pod \"38fb8aa4-6e7a-4570-904b-579022f834d7\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " Nov 03 04:15:45.792068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.792022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\" (UniqueName: \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\") pod \"38fb8aa4-6e7a-4570-904b-579022f834d7\" (UID: \"38fb8aa4-6e7a-4570-904b-579022f834d7\") " Nov 03 04:15:45.792306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.792279 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume" pod "38fb8aa4-6e7a-4570-904b-579022f834d7" (UID: "38fb8aa4-6e7a-4570-904b-579022f834d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:45.792494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.792469 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fb8aa4-6e7a-4570-904b-579022f834d7-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:45.794268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.794251 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4" pod "38fb8aa4-6e7a-4570-904b-579022f834d7" (UID: "38fb8aa4-6e7a-4570-904b-579022f834d7"). InnerVolumeSpecName "kube-api-access-76sb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:45.893200 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.893099 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\") pod \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " Nov 03 04:15:45.893372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.893271 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\" (UniqueName: \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\") pod \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\" (UID: \"cfd810c6-10f1-4b67-837f-a7d3e0858e6c\") " Nov 03 04:15:45.893618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.893573 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume" pod "cfd810c6-10f1-4b67-837f-a7d3e0858e6c" (UID: "cfd810c6-10f1-4b67-837f-a7d3e0858e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:45.893746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.893727 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:45.893814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.893750 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sb4\" (UniqueName: \"kubernetes.io/projected/38fb8aa4-6e7a-4570-904b-579022f834d7-kube-api-access-76sb4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:45.895714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.895689 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt" pod "cfd810c6-10f1-4b67-837f-a7d3e0858e6c" (UID: "cfd810c6-10f1-4b67-837f-a7d3e0858e6c"). InnerVolumeSpecName "kube-api-access-nlqrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:45.995305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:45.995267 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqrt\" (UniqueName: \"kubernetes.io/projected/cfd810c6-10f1-4b67-837f-a7d3e0858e6c-kube-api-access-nlqrt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:46.557030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.556989 5275 generic.go:356] "Generic (PLEG): container finished" podID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" containerID="6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a" exitCode=0 Nov 03 04:15:46.557528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.557051 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" Nov 03 04:15:46.557528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.557073 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" event={"ID":"cfd810c6-10f1-4b67-837f-a7d3e0858e6c","Type":"ContainerDied","Data":"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a"} Nov 03 04:15:46.557528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.557139 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc" event={"ID":"cfd810c6-10f1-4b67-837f-a7d3e0858e6c","Type":"ContainerDied","Data":"8f1839a1aa35f69ee22d7b3ca476d7d73f4b3337909d2747b5008d4c0f6e15c3"} Nov 03 04:15:46.557528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.557170 5275 scope.go:122] "RemoveContainer" containerID="6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a" Nov 03 04:15:46.559150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.559036 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" event={"ID":"38fb8aa4-6e7a-4570-904b-579022f834d7","Type":"ContainerDied","Data":"0ac685e9eaf0ff113c9acff0426ed2f65f4db5cf14d38a3fbc439834bb1b62ed"} Nov 03 04:15:46.559150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.559144 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g" Nov 03 04:15:46.570619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.570596 5275 scope.go:122] "RemoveContainer" containerID="6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a" Nov 03 04:15:46.570989 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:15:46.570965 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a\": container with ID starting with 6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a not found: ID does not exist" containerID="6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a" Nov 03 04:15:46.571086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.571002 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a"} err="failed to get container status \"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a\": rpc error: code = NotFound desc = could not find container \"6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a\": container with ID starting with 6971986f80d5730b2a98808f11d94b1198eb9926fe1419184af99fb1725dd06a not found: ID does not exist" Nov 03 04:15:46.571086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.571026 5275 scope.go:122] "RemoveContainer" containerID="f043e1704daa3804cccd5a8eb9e04936c7b759ced25844e4217933c26ccfddbb" Nov 03 04:15:46.724499 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.724417 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:15:46.730135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.730110 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8540f6-58db4b8f6f-djp2g"] Nov 03 04:15:46.839137 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.839094 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:15:46.842901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.842863 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8540f6-64b4d98bd7-9wqqc"] Nov 03 04:15:46.985939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.985835 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fb8aa4-6e7a-4570-904b-579022f834d7" path="/var/lib/kubelet/pods/38fb8aa4-6e7a-4570-904b-579022f834d7/volumes" Nov 03 04:15:46.986333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:46.986319 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" path="/var/lib/kubelet/pods/cfd810c6-10f1-4b67-837f-a7d3e0858e6c/volumes" Nov 03 04:15:50.427188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.427148 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:15:50.428049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428027 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.428049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428049 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428069 5275 cpu_manager.go:414] "containerMap: removing container" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428078 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428088 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428096 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428110 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428141 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428151 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428159 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428176 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428183 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428195 5275 cpu_manager.go:414] "containerMap: removing container" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428203 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428213 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428220 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428230 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428237 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428247 5275 cpu_manager.go:414] "containerMap: removing container" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428254 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.428257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428267 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428274 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428286 5275 cpu_manager.go:414] "containerMap: removing container" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428293 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428304 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428311 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428326 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428333 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428344 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" containerName="payload-pre-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428352 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" containerName="payload-pre-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428369 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428375 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428385 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428392 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428402 5275 cpu_manager.go:414] "containerMap: removing container" podUID="38fb8aa4-6e7a-4570-904b-579022f834d7" containerName="payload-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428408 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fb8aa4-6e7a-4570-904b-579022f834d7" containerName="payload-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428421 5275 cpu_manager.go:414] "containerMap: removing container" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428427 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428439 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428447 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428460 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428466 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428476 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428483 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428493 5275 cpu_manager.go:414] "containerMap: removing container" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428500 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428513 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428520 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428533 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428540 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428556 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428563 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428574 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428581 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428604 5275 cpu_manager.go:414] "containerMap: removing container" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428610 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428634 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428641 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428652 5275 cpu_manager.go:414] "containerMap: removing container" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428658 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428804 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428817 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428828 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428837 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428866 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428875 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428885 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8b3b08-1613-46ad-b053-2634b4d7a396" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428893 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428904 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428913 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428921 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428929 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428938 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428948 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428959 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428968 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428978 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428986 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.428994 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429005 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429015 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429025 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429034 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429043 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f644c466-1b9c-495d-b9d8-193c22509b2e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429051 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429061 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429070 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfded98-72c8-4567-b05c-e5a1386f8f1a" containerName="istio-proxy" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429080 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd810c6-10f1-4b67-837f-a7d3e0858e6c" containerName="payload-pre-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429089 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a81132-06bb-410d-b805-00be6c599442" containerName="cleanup" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429099 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fb8aa4-6e7a-4570-904b-579022f834d7" containerName="payload-processing" Nov 03 04:15:50.429315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.429111 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:50.434430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.434408 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.508353 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.508321 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3c1cec-openshift-default-dockercfg-hqt69\"" Nov 03 04:15:50.566916 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.566877 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.567277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.567254 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl28t\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.567517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.567500 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.567704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.567688 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.567997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.567977 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.568237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.568218 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.568453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.568435 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.568699 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.568681 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.568931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.568915 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.616870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.611225 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:15:50.672107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.672049 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.673129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.672884 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.673446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.673412 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.674032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674001 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.673951 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674698 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.673671 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674758 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674902 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl28t\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674957 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.674991 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.675362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.675324 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.676046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.675369 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.676046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.675783 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.677899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.677816 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.678341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.678317 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.733999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.733966 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.738199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.738163 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl28t\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\") pod \"e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:50.748869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:50.748821 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:51.190812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.190780 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:15:51.192561 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:51.192524 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cffde05_c4db_4390_b877_2bca581764da.slice/crio-7b9f70281ea7c20a07393905eb0b02e665df273e0725fc8dbec5a31dc8845364 WatchSource:0}: Error finding container 7b9f70281ea7c20a07393905eb0b02e665df273e0725fc8dbec5a31dc8845364: Status 404 returned error can't find the container with id 7b9f70281ea7c20a07393905eb0b02e665df273e0725fc8dbec5a31dc8845364 Nov 03 04:15:51.195517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.195483 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:15:51.195613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.195554 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:15:51.195613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.195585 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:15:51.616392 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.616357 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerStarted","Data":"649d7460bd0c49ada3e20bb0942d3b422b347823e830b7989e698614f3ef066e"} Nov 03 04:15:51.616954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.616411 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerStarted","Data":"7b9f70281ea7c20a07393905eb0b02e665df273e0725fc8dbec5a31dc8845364"} Nov 03 04:15:51.741191 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.741139 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podStartSLOduration=1.7411234260000001 podStartE2EDuration="1.741123426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:51.73692683 +0000 UTC m=+4442.889615258" watchObservedRunningTime="2026-11-03 04:15:51.741123426 +0000 UTC m=+4442.893811846" Nov 03 04:15:51.749396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:51.749356 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:52.756367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:52.756337 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:53.632205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:53.632170 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:53.633389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:53.633366 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:15:58.518809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.518774 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:15:58.519404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519390 5275 cpu_manager.go:414] "containerMap: removing container" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519405 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519425 5275 cpu_manager.go:414] "containerMap: removing container" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519431 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519436 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519441 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f63c44a-1883-4b56-b061-4ec2128a1ea2" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519449 5275 cpu_manager.go:414] "containerMap: removing container" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:58.519459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519453 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9e270c-cfea-48a8-a370-72ad9d72884e" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519462 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519467 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519478 5275 cpu_manager.go:414] "containerMap: removing container" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519482 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519497 5275 cpu_manager.go:414] "containerMap: removing container" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519501 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519602 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be1a790-6445-4455-875b-3a2417ecc3c7" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519612 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c01e88-5156-4a40-96bc-727cedb46c81" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519620 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c75986-6ac3-456f-b746-6fc49e78f9d5" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519631 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1452961-d9b1-4411-ad3b-74b96983de5f" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519638 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fd1bf8-d2a5-4151-ad29-8cf28dcf5000" containerName="cleanup" Nov 03 04:15:58.520118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.519644 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c94ec6-28aa-4898-8b15-3d13e20b05e2" containerName="cleanup" Nov 03 04:15:58.525117 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.525094 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.580604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.580570 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:15:58.600068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.599929 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5g5s\" (UniqueName: \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.600957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.600807 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.630872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.630195 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3c1cec-dockercfg-xpl6w\"" Nov 03 04:15:58.630872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.630301 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3c1cec\"" Nov 03 04:15:58.728462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.719198 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.728462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.719508 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5g5s\" (UniqueName: \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.728462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.722178 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.728462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.724115 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:15:58.731067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.730650 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:15:58.731067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.730740 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.781885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.780769 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5g5s\" (UniqueName: \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\") pod \"payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.821414 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.821372 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.821611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.821552 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25l6\" (UniqueName: \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.840397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.840357 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:58.922997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.922956 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.923158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.923131 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25l6\" (UniqueName: \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.923979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.923947 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:58.989552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:58.989516 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25l6\" (UniqueName: \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\") pod \"payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:59.072976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.072873 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:59.225909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.225876 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:15:59.226647 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:59.226625 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31040ac2_4722_4c7f_80e1_c96772349c66.slice/crio-715381a901db16acfe91176f141990b1935267a46d15c504176c52c949bb45d8 WatchSource:0}: Error finding container 715381a901db16acfe91176f141990b1935267a46d15c504176c52c949bb45d8: Status 404 returned error can't find the container with id 715381a901db16acfe91176f141990b1935267a46d15c504176c52c949bb45d8 Nov 03 04:15:59.458064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.458024 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:15:59.459263 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:15:59.459235 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafea22a_939c_4781_b06d_1def3a355682.slice/crio-712c9927b64c55bc8d96981c946d08f032683f0516afaa3395ae3f8c284da03e WatchSource:0}: Error finding container 712c9927b64c55bc8d96981c946d08f032683f0516afaa3395ae3f8c284da03e: Status 404 returned error can't find the container with id 712c9927b64c55bc8d96981c946d08f032683f0516afaa3395ae3f8c284da03e Nov 03 04:15:59.768380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.768273 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" event={"ID":"31040ac2-4722-4c7f-80e1-c96772349c66","Type":"ContainerStarted","Data":"49fad223774bb578df683a477a698036146074a137959473c46539772d88104a"} Nov 03 04:15:59.768380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.768347 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:15:59.768380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.768371 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" event={"ID":"31040ac2-4722-4c7f-80e1-c96772349c66","Type":"ContainerStarted","Data":"715381a901db16acfe91176f141990b1935267a46d15c504176c52c949bb45d8"} Nov 03 04:15:59.770319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.770288 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" event={"ID":"eafea22a-939c-4781-b06d-1def3a355682","Type":"ContainerStarted","Data":"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e"} Nov 03 04:15:59.770465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.770350 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:15:59.770465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.770371 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" event={"ID":"eafea22a-939c-4781-b06d-1def3a355682","Type":"ContainerStarted","Data":"712c9927b64c55bc8d96981c946d08f032683f0516afaa3395ae3f8c284da03e"} Nov 03 04:15:59.885179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:15:59.885122 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" podStartSLOduration=1.885105466 podStartE2EDuration="1.885105466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:59.884866385 +0000 UTC m=+4451.037554788" watchObservedRunningTime="2026-11-03 04:15:59.885105466 +0000 UTC m=+4451.037793887" Nov 03 04:16:00.008018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:00.007264 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" podStartSLOduration=2.007238091 podStartE2EDuration="2.007238091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:16:00.001665406 +0000 UTC m=+4451.154353885" watchObservedRunningTime="2026-11-03 04:16:00.007238091 +0000 UTC m=+4451.159926516" Nov 03 04:16:02.331294 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:16:02.331257 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:16:10.782610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:10.782573 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:16:10.783134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:10.782657 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:16:29.321338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.321299 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx"] Nov 03 04:16:29.325763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.325739 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.422421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.422378 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.422631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.422519 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjch\" (UniqueName: \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.422631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.422568 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.428015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.427989 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:16:29.428269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.428253 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:16:29.433220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.433191 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx"] Nov 03 04:16:29.467683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.467650 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:16:29.467928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.467754 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:16:29.524605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.524571 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.524835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.524634 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjch\" (UniqueName: \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.524835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.524659 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.525460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.525440 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.527257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.527236 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.586049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.585972 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjch\" (UniqueName: \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\") pod \"maas-api-revoke-sub-6335e74f-st2cx\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:29.640473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:29.640435 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:30.030651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:30.030624 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx"] Nov 03 04:16:30.032367 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:16:30.032339 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda536e075_1629_4612_a9ac_492c20c7e6c9.slice/crio-72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108 WatchSource:0}: Error finding container 72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108: Status 404 returned error can't find the container with id 72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108 Nov 03 04:16:31.018618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:31.018584 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" event={"ID":"a536e075-1629-4612-a9ac-492c20c7e6c9","Type":"ContainerStarted","Data":"ebb9c84a14087cc6cca1c9ea9ecd8357b50f1a585b4d6517f98ac503c63b8a12"} Nov 03 04:16:31.018618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:31.018633 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" event={"ID":"a536e075-1629-4612-a9ac-492c20c7e6c9","Type":"ContainerStarted","Data":"72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108"} Nov 03 04:16:31.140042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:31.139989 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" podStartSLOduration=2.139974883 podStartE2EDuration="2.139974883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:16:31.13732501 +0000 UTC m=+4482.290013430" watchObservedRunningTime="2026-11-03 04:16:31.139974883 +0000 UTC m=+4482.292663304" Nov 03 04:16:32.034369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:32.034329 5275 generic.go:356] "Generic (PLEG): container finished" podID="a536e075-1629-4612-a9ac-492c20c7e6c9" containerID="ebb9c84a14087cc6cca1c9ea9ecd8357b50f1a585b4d6517f98ac503c63b8a12" exitCode=0 Nov 03 04:16:32.034893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:32.034488 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" event={"ID":"a536e075-1629-4612-a9ac-492c20c7e6c9","Type":"ContainerDied","Data":"ebb9c84a14087cc6cca1c9ea9ecd8357b50f1a585b4d6517f98ac503c63b8a12"} Nov 03 04:16:32.331017 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:16:32.330928 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:16:33.199222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.199194 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:33.286229 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.286192 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\") pod \"a536e075-1629-4612-a9ac-492c20c7e6c9\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " Nov 03 04:16:33.286449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.286272 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\") pod \"a536e075-1629-4612-a9ac-492c20c7e6c9\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " Nov 03 04:16:33.286449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.286309 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\" (UniqueName: \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\") pod \"a536e075-1629-4612-a9ac-492c20c7e6c9\" (UID: \"a536e075-1629-4612-a9ac-492c20c7e6c9\") " Nov 03 04:16:33.286659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.286630 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca" pod "a536e075-1629-4612-a9ac-492c20c7e6c9" (UID: "a536e075-1629-4612-a9ac-492c20c7e6c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:16:33.287073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.287049 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:33.288509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.288480 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca" pod "a536e075-1629-4612-a9ac-492c20c7e6c9" (UID: "a536e075-1629-4612-a9ac-492c20c7e6c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:16:33.288608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.288571 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch" pod "a536e075-1629-4612-a9ac-492c20c7e6c9" (UID: "a536e075-1629-4612-a9ac-492c20c7e6c9"). InnerVolumeSpecName "kube-api-access-znjch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:16:33.396162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.396117 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a536e075-1629-4612-a9ac-492c20c7e6c9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:33.396439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:33.396419 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znjch\" (UniqueName: \"kubernetes.io/projected/a536e075-1629-4612-a9ac-492c20c7e6c9-kube-api-access-znjch\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:34.051340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:34.051311 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" Nov 03 04:16:34.051545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:34.051347 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx" event={"ID":"a536e075-1629-4612-a9ac-492c20c7e6c9","Type":"ContainerDied","Data":"72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108"} Nov 03 04:16:34.051545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:34.051400 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72da8080681831cb8a2aac305d5983d18160c6cfe740931a699df297184e3108" Nov 03 04:16:45.154989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.154951 5275 generic.go:356] "Generic (PLEG): container finished" podID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerID="78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50" exitCode=0 Nov 03 04:16:45.155473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.155023 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerDied","Data":"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50"} Nov 03 04:16:45.155533 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.155483 5275 scope.go:122] "RemoveContainer" containerID="78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50" Nov 03 04:16:45.157579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.157545 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:16:45.158034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.157631 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:16:45.158034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:45.157671 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:16:46.188886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:46.188812 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerStarted","Data":"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390"} Nov 03 04:16:47.138238 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:47.138201 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:16:47.139609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:47.139578 5275 patch_prober.go:29] interesting pod/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 04:16:47.139754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:47.139645 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 04:16:47.729807 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:47.729770 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:16:48.138272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:48.138227 5275 patch_prober.go:29] interesting pod/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 04:16:48.138464 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:48.138303 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 04:16:49.138243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:49.138202 5275 patch_prober.go:29] interesting pod/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 04:16:49.138676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:49.138278 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 04:16:50.137805 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:50.137761 5275 patch_prober.go:29] interesting pod/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 04:16:50.138055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:50.137830 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 04:16:50.396361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:50.396284 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:16:50.432174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:50.432147 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:16:51.138595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:51.138555 5275 patch_prober.go:29] interesting pod/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 04:16:51.138789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:51.138611 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 04:16:52.142924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:52.142893 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:16:52.144066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:52.144046 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:16:54.104329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:54.104292 5275 scope.go:122] "RemoveContainer" containerID="70dd152f18d8504996a52031a5f3be42197039d4af416177ed78bf9c39ebf951" Nov 03 04:16:54.115332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:54.115308 5275 scope.go:122] "RemoveContainer" containerID="dfd9492b2667da5529c5e42fdc82c06900dcf569a3b0c9cc4b6c6130afb3e6d7" Nov 03 04:16:59.245751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:59.245715 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:16:59.246262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:59.245944 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" containerID="cri-o://71adf17ecef6fb03acb718447527c89ab05e0064223eba97c3371de813669eaf" gracePeriod=30 Nov 03 04:16:59.572405 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:59.572370 5275 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:16:59.572598 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:16:59.572433 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:17:02.331451 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:17:02.331410 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:17:04.457578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.456908 5275 generic.go:356] "Generic (PLEG): container finished" podID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerID="71adf17ecef6fb03acb718447527c89ab05e0064223eba97c3371de813669eaf" exitCode=0 Nov 03 04:17:04.457578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.457208 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerDied","Data":"71adf17ecef6fb03acb718447527c89ab05e0064223eba97c3371de813669eaf"} Nov 03 04:17:04.457578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.457275 5275 scope.go:122] "RemoveContainer" containerID="a18639815908874aa320ef5ae8843367e37fc745547e534af3cfb012d342d6a6" Nov 03 04:17:04.560049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.560022 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:17:04.634947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.634842 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635000 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635037 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635062 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635156 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635273 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635302 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635330 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635324 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:04.635585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635396 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:17:04.635585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635418 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\") pod \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\" (UID: \"b1d45890-5837-4fd6-a52b-abce2f3b59c3\") " Nov 03 04:17:04.635585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635421 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:04.635585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635573 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:04.635989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.635649 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:04.636164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.636024 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.636164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.636045 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.636164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.636059 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.636164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.636072 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.636164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.636084 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.637819 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.637789 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:04.637972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.637909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "kube-api-access-jmpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:04.638214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.638190 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:04.638214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.638202 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo" pod "b1d45890-5837-4fd6-a52b-abce2f3b59c3" (UID: "b1d45890-5837-4fd6-a52b-abce2f3b59c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:17:04.737686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.737648 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.737686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.737684 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.737686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.737694 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d45890-5837-4fd6-a52b-abce2f3b59c3-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:04.737980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:04.737704 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpz2\" (UniqueName: \"kubernetes.io/projected/b1d45890-5837-4fd6-a52b-abce2f3b59c3-kube-api-access-jmpz2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:05.465371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:05.465336 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" Nov 03 04:17:05.465371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:05.465351 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2" event={"ID":"b1d45890-5837-4fd6-a52b-abce2f3b59c3","Type":"ContainerDied","Data":"96deb5f12b4a0b24cb0c00d3c4de6bb72e70cca80a4fb63f78d4cd094c0a523b"} Nov 03 04:17:05.465913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:05.465417 5275 scope.go:122] "RemoveContainer" containerID="71adf17ecef6fb03acb718447527c89ab05e0064223eba97c3371de813669eaf" Nov 03 04:17:05.680491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:05.680454 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:17:05.729891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:05.729797 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-276h2"] Nov 03 04:17:06.988531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:06.988493 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" path="/var/lib/kubelet/pods/b1d45890-5837-4fd6-a52b-abce2f3b59c3/volumes" Nov 03 04:17:32.330769 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:17:32.330723 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:17:37.229486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.229437 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7"] Nov 03 04:17:37.230463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230442 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.230517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230466 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.230517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230489 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a536e075-1629-4612-a9ac-492c20c7e6c9" containerName="revoke-keys" Nov 03 04:17:37.230517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230497 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a536e075-1629-4612-a9ac-492c20c7e6c9" containerName="revoke-keys" Nov 03 04:17:37.230517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230510 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.230652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230517 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.230652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230640 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.230719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230654 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a536e075-1629-4612-a9ac-492c20c7e6c9" containerName="revoke-keys" Nov 03 04:17:37.230719 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.230665 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d45890-5837-4fd6-a52b-abce2f3b59c3" containerName="istio-proxy" Nov 03 04:17:37.234929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.234911 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.288945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.288906 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7"] Nov 03 04:17:37.333944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.333911 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:17:37.334211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.333948 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:17:37.334211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.334028 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:17:37.336380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.336361 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:17:37.374615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.374572 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.374812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.374750 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ktz\" (UniqueName: \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.374988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.374970 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.475890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.475820 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ktz\" (UniqueName: \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.476080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.476018 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.476164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.476124 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.477008 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.476978 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.478800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.478769 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.537185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.537146 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ktz\" (UniqueName: \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\") pod \"maas-api-revoke-sub-fd7093c7-tj4j7\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.549094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.549061 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:37.936524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:37.936493 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7"] Nov 03 04:17:37.938340 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:17:37.938313 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdaf8892_2dc4_45f5_8212_5dd44ebbb48e.slice/crio-f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970 WatchSource:0}: Error finding container f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970: Status 404 returned error can't find the container with id f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970 Nov 03 04:17:38.743900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:38.743834 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" event={"ID":"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e","Type":"ContainerStarted","Data":"43ff70a08bb5688bd46fab0c5decd2ef5392ff47c5cf50f5d32f947930f007e8"} Nov 03 04:17:38.744413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:38.743916 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" event={"ID":"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e","Type":"ContainerStarted","Data":"f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970"} Nov 03 04:17:38.860478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:38.860425 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" podStartSLOduration=1.8604103520000002 podStartE2EDuration="1.860410352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:17:38.858795734 +0000 UTC m=+4550.011484153" watchObservedRunningTime="2026-11-03 04:17:38.860410352 +0000 UTC m=+4550.013098772" Nov 03 04:17:39.752546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:39.752507 5275 generic.go:356] "Generic (PLEG): container finished" podID="fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" containerID="43ff70a08bb5688bd46fab0c5decd2ef5392ff47c5cf50f5d32f947930f007e8" exitCode=0 Nov 03 04:17:39.753047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:39.752594 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" event={"ID":"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e","Type":"ContainerDied","Data":"43ff70a08bb5688bd46fab0c5decd2ef5392ff47c5cf50f5d32f947930f007e8"} Nov 03 04:17:40.946111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:40.946079 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:41.021045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.021016 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\") pod \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " Nov 03 04:17:41.021221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.021123 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\" (UniqueName: \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\") pod \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " Nov 03 04:17:41.021221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.021151 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\") pod \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\" (UID: \"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e\") " Nov 03 04:17:41.021572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.021539 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca" pod "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" (UID: "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:17:41.023415 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.023390 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca" pod "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" (UID: "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:17:41.023523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.023417 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz" pod "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" (UID: "fdaf8892-2dc4-45f5-8212-5dd44ebbb48e"). InnerVolumeSpecName "kube-api-access-72ktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:41.122893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.122859 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ktz\" (UniqueName: \"kubernetes.io/projected/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-kube-api-access-72ktz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:41.122893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.122891 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:41.122893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.122902 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:41.629120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.629080 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 04:17:41.645068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.645033 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 04:17:41.774896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.774551 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" Nov 03 04:17:41.774896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.774576 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7" event={"ID":"fdaf8892-2dc4-45f5-8212-5dd44ebbb48e","Type":"ContainerDied","Data":"f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970"} Nov 03 04:17:41.774896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:41.774642 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f05746110ab1e49af3b2d35e0fc75180f8a4ba21f5c635dbb54009c2a1bb3970" Nov 03 04:17:53.878035 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.877994 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5"] Nov 03 04:17:53.879067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.879041 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" containerName="revoke-keys" Nov 03 04:17:53.879067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.879068 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" containerName="revoke-keys" Nov 03 04:17:53.879309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.879293 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" containerName="revoke-keys" Nov 03 04:17:53.883045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.883023 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:53.936259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.936221 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5"] Nov 03 04:17:53.979748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.979696 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:17:53.979748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.979728 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:17:53.980152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.979753 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:17:53.985493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:53.985461 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:17:54.032600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.032565 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.032600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.032609 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.032942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.032790 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjjt\" (UniqueName: \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.134154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.134062 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjjt\" (UniqueName: \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.134335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.134230 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.134335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.134265 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.134965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.134942 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.136825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.136807 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.196608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.196571 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjjt\" (UniqueName: \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\") pod \"maas-api-revoke-sub-ca41d644-lxgt5\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.199980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.199954 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:54.622027 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.621980 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5"] Nov 03 04:17:54.907162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:54.907127 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" event={"ID":"686c1465-24c0-4f0d-95d6-22ef678caa63","Type":"ContainerStarted","Data":"71922d16597f013e2eab24307dc6d1dd5478bccc05932d5050efaf8d61d4cc83"} Nov 03 04:17:55.648599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:55.648566 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 04:17:55.916433 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:55.916345 5275 generic.go:356] "Generic (PLEG): container finished" podID="686c1465-24c0-4f0d-95d6-22ef678caa63" containerID="c1d47631cb362dbb532b836b82080eb11261bdd5c98c5973d3086e0b63dd723d" exitCode=0 Nov 03 04:17:55.916830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:55.916487 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" event={"ID":"686c1465-24c0-4f0d-95d6-22ef678caa63","Type":"ContainerDied","Data":"c1d47631cb362dbb532b836b82080eb11261bdd5c98c5973d3086e0b63dd723d"} Nov 03 04:17:56.651223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.651185 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" containerID="cri-o://19371ecc43380e4df31eafb37211f1839c926eb6cc3e13ef7cb4a910ce5f3b9e" gracePeriod=45 Nov 03 04:17:56.669504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.669462 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" containerID="cri-o://d91a840355f9ed9107fddacc8f59357348acca13c885153437ac45f23186e69b" gracePeriod=45 Nov 03 04:17:56.929654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.929558 5275 generic.go:356] "Generic (PLEG): container finished" podID="99758ef7-4c84-451c-ba20-8b437d442785" containerID="19371ecc43380e4df31eafb37211f1839c926eb6cc3e13ef7cb4a910ce5f3b9e" exitCode=0 Nov 03 04:17:56.930222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.929711 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" event={"ID":"99758ef7-4c84-451c-ba20-8b437d442785","Type":"ContainerDied","Data":"19371ecc43380e4df31eafb37211f1839c926eb6cc3e13ef7cb4a910ce5f3b9e"} Nov 03 04:17:56.933463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.933121 5275 generic.go:356] "Generic (PLEG): container finished" podID="027de7ab-0a79-45fd-afce-1dee74384a53" containerID="d91a840355f9ed9107fddacc8f59357348acca13c885153437ac45f23186e69b" exitCode=0 Nov 03 04:17:56.933463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.933158 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" event={"ID":"027de7ab-0a79-45fd-afce-1dee74384a53","Type":"ContainerDied","Data":"d91a840355f9ed9107fddacc8f59357348acca13c885153437ac45f23186e69b"} Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.991248 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.995907 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\" (UniqueName: \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.995987 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.996060 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.996114 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.996169 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.998631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.996322 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\" (UniqueName: \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\") pod \"99758ef7-4c84-451c-ba20-8b437d442785\" (UID: \"99758ef7-4c84-451c-ba20-8b437d442785\") " Nov 03 04:17:56.999207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.998674 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:56.999207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.998838 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.003823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:56.999692 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.003823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.000626 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "kube-api-access-4pmvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:57.003823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.002223 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:17:57.003823 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.003609 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm" pod "99758ef7-4c84-451c-ba20-8b437d442785" (UID: "99758ef7-4c84-451c-ba20-8b437d442785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113586 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99758ef7-4c84-451c-ba20-8b437d442785-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113633 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmvt\" (UniqueName: \"kubernetes.io/projected/99758ef7-4c84-451c-ba20-8b437d442785-kube-api-access-4pmvt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113649 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113663 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113676 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.117535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.113690 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99758ef7-4c84-451c-ba20-8b437d442785-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.164335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.164301 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 04:17:57.214380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214260 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214313 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\" (UniqueName: \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214368 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214406 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214547 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214578 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\" (UniqueName: \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\") pod \"027de7ab-0a79-45fd-afce-1dee74384a53\" (UID: \"027de7ab-0a79-45fd-afce-1dee74384a53\") " Nov 03 04:17:57.214712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214644 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.215025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.214758 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.215111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.215087 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.215261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.215246 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.215302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.215265 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.215302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.215278 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.218839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.218806 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:17:57.219297 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.219266 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "kube-api-access-q7cwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:57.220725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.220704 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm" pod "027de7ab-0a79-45fd-afce-1dee74384a53" (UID: "027de7ab-0a79-45fd-afce-1dee74384a53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:57.250867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.248809 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.315764 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\") pod \"686c1465-24c0-4f0d-95d6-22ef678caa63\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.315933 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\") pod \"686c1465-24c0-4f0d-95d6-22ef678caa63\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316037 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\" (UniqueName: \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\") pod \"686c1465-24c0-4f0d-95d6-22ef678caa63\" (UID: \"686c1465-24c0-4f0d-95d6-22ef678caa63\") " Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316299 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca" pod "686c1465-24c0-4f0d-95d6-22ef678caa63" (UID: "686c1465-24c0-4f0d-95d6-22ef678caa63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316622 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027de7ab-0a79-45fd-afce-1dee74384a53-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316644 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316660 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cwh\" (UniqueName: \"kubernetes.io/projected/027de7ab-0a79-45fd-afce-1dee74384a53-kube-api-access-q7cwh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.316673 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027de7ab-0a79-45fd-afce-1dee74384a53-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.320226 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt" pod "686c1465-24c0-4f0d-95d6-22ef678caa63" (UID: "686c1465-24c0-4f0d-95d6-22ef678caa63"). InnerVolumeSpecName "kube-api-access-pfjjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:57.322895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.322095 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca" pod "686c1465-24c0-4f0d-95d6-22ef678caa63" (UID: "686c1465-24c0-4f0d-95d6-22ef678caa63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:17:57.420874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.419024 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjjt\" (UniqueName: \"kubernetes.io/projected/686c1465-24c0-4f0d-95d6-22ef678caa63-kube-api-access-pfjjt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.420874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.419058 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/686c1465-24c0-4f0d-95d6-22ef678caa63-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.944918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.944579 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" event={"ID":"99758ef7-4c84-451c-ba20-8b437d442785","Type":"ContainerDied","Data":"a037ff314ba475c0615dcff94bafe049501d6ba768cd21139805ce07273b98ed"} Nov 03 04:17:57.944918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.944648 5275 scope.go:122] "RemoveContainer" containerID="19371ecc43380e4df31eafb37211f1839c926eb6cc3e13ef7cb4a910ce5f3b9e" Nov 03 04:17:57.944918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.944589 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p" Nov 03 04:17:57.948608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.948575 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" event={"ID":"027de7ab-0a79-45fd-afce-1dee74384a53","Type":"ContainerDied","Data":"c40a6dc1a6c65150b47e14ef1fb93a96f846cbfaa136531f0f7120e4fce35903"} Nov 03 04:17:57.948767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.948674 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9" Nov 03 04:17:57.951348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.951318 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" event={"ID":"686c1465-24c0-4f0d-95d6-22ef678caa63","Type":"ContainerDied","Data":"71922d16597f013e2eab24307dc6d1dd5478bccc05932d5050efaf8d61d4cc83"} Nov 03 04:17:57.951477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.951370 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71922d16597f013e2eab24307dc6d1dd5478bccc05932d5050efaf8d61d4cc83" Nov 03 04:17:57.951477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.951429 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5" Nov 03 04:17:57.959965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:57.959913 5275 scope.go:122] "RemoveContainer" containerID="d91a840355f9ed9107fddacc8f59357348acca13c885153437ac45f23186e69b" Nov 03 04:17:58.112618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.112578 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 04:17:58.117964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.117934 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/simulator-w3-68b670-kserve-6bd75776f7-nvww9"] Nov 03 04:17:58.222838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.222739 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 04:17:58.228341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.228306 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/premium-w3-68b670-kserve-7fb796b996-qvl9p"] Nov 03 04:17:58.986998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.986960 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" path="/var/lib/kubelet/pods/027de7ab-0a79-45fd-afce-1dee74384a53/volumes" Nov 03 04:17:58.987488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:58.987477 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99758ef7-4c84-451c-ba20-8b437d442785" path="/var/lib/kubelet/pods/99758ef7-4c84-451c-ba20-8b437d442785/volumes" Nov 03 04:17:59.970821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.970735 5275 generic.go:356] "Generic (PLEG): container finished" podID="7cffde05-c4db-4390-b877-2bca581764da" containerID="649d7460bd0c49ada3e20bb0942d3b422b347823e830b7989e698614f3ef066e" exitCode=0 Nov 03 04:17:59.970821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.970802 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerDied","Data":"649d7460bd0c49ada3e20bb0942d3b422b347823e830b7989e698614f3ef066e"} Nov 03 04:17:59.971268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.971254 5275 scope.go:122] "RemoveContainer" containerID="649d7460bd0c49ada3e20bb0942d3b422b347823e830b7989e698614f3ef066e" Nov 03 04:17:59.972956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.972929 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:17:59.973020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.973000 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:17:59.973058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:17:59.973032 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:18:00.663186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.662926 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" podUID="29580c82-d705-4e84-a050-69f3610bec2c" containerName="payload-processing" containerID="cri-o://cb76596e0794a64e81c485273d4e851b132dc87a308348166c3a05ca252dca83" gracePeriod=25 Nov 03 04:18:00.996031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.995998 5275 generic.go:356] "Generic (PLEG): container finished" podID="29580c82-d705-4e84-a050-69f3610bec2c" containerID="cb76596e0794a64e81c485273d4e851b132dc87a308348166c3a05ca252dca83" exitCode=0 Nov 03 04:18:00.996213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.996069 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" event={"ID":"29580c82-d705-4e84-a050-69f3610bec2c","Type":"ContainerDied","Data":"cb76596e0794a64e81c485273d4e851b132dc87a308348166c3a05ca252dca83"} Nov 03 04:18:00.996213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.996120 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" event={"ID":"29580c82-d705-4e84-a050-69f3610bec2c","Type":"ContainerDied","Data":"cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531"} Nov 03 04:18:00.996213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.996137 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6d31385508dbc512b8023197c08d64f5c17aad1aa8de7f288db5dee4991531" Nov 03 04:18:00.998701 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.998679 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerStarted","Data":"3de9939faf79c3affc37943b204635ed8c8ab8bf4c1ffeec4b19d7a5570985ce"} Nov 03 04:18:00.999322 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:00.999271 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 04:18:01.095518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.095478 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\" (UniqueName: \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\") pod \"29580c82-d705-4e84-a050-69f3610bec2c\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " Nov 03 04:18:01.095752 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.095621 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\") pod \"29580c82-d705-4e84-a050-69f3610bec2c\" (UID: \"29580c82-d705-4e84-a050-69f3610bec2c\") " Nov 03 04:18:01.096015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.095982 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume" pod "29580c82-d705-4e84-a050-69f3610bec2c" (UID: "29580c82-d705-4e84-a050-69f3610bec2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:18:01.096306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.096288 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29580c82-d705-4e84-a050-69f3610bec2c-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:18:01.098377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.098348 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz" pod "29580c82-d705-4e84-a050-69f3610bec2c" (UID: "29580c82-d705-4e84-a050-69f3610bec2c"). InnerVolumeSpecName "kube-api-access-brmbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:18:01.198910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.198834 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmbz\" (UniqueName: \"kubernetes.io/projected/29580c82-d705-4e84-a050-69f3610bec2c-kube-api-access-brmbz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:18:01.768874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.765307 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:18:01.768874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.765451 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 04:18:01.768874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:01.765495 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 04:18:02.007406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.007376 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h" Nov 03 04:18:02.184431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.184389 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 04:18:02.233247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.233209 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-68b670-6b58c994c5-wts8h"] Nov 03 04:18:02.331669 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:18:02.331634 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:18:02.753351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.753309 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 04:18:02.753669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.753633 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 04:18:02.996219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:02.996176 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29580c82-d705-4e84-a050-69f3610bec2c" path="/var/lib/kubelet/pods/29580c82-d705-4e84-a050-69f3610bec2c/volumes" Nov 03 04:18:03.759882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.758974 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 04:18:03.759882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.759041 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 04:18:03.882602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.882563 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:18:03.883569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883545 5275 cpu_manager.go:414] "containerMap: removing container" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" Nov 03 04:18:03.883666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883573 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" Nov 03 04:18:03.883666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883592 5275 cpu_manager.go:414] "containerMap: removing container" podUID="686c1465-24c0-4f0d-95d6-22ef678caa63" containerName="revoke-keys" Nov 03 04:18:03.883666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883601 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686c1465-24c0-4f0d-95d6-22ef678caa63" containerName="revoke-keys" Nov 03 04:18:03.883666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883619 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29580c82-d705-4e84-a050-69f3610bec2c" containerName="payload-processing" Nov 03 04:18:03.883666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883630 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29580c82-d705-4e84-a050-69f3610bec2c" containerName="payload-processing" Nov 03 04:18:03.883884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883679 5275 cpu_manager.go:414] "containerMap: removing container" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" Nov 03 04:18:03.883884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.883686 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" Nov 03 04:18:03.884941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.884920 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="99758ef7-4c84-451c-ba20-8b437d442785" containerName="main" Nov 03 04:18:03.885013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.884953 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="686c1465-24c0-4f0d-95d6-22ef678caa63" containerName="revoke-keys" Nov 03 04:18:03.885013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.884968 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29580c82-d705-4e84-a050-69f3610bec2c" containerName="payload-processing" Nov 03 04:18:03.885013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.884993 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="027de7ab-0a79-45fd-afce-1dee74384a53" containerName="main" Nov 03 04:18:03.887955 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.887934 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938301 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjftf\" (UniqueName: \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938370 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938438 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938575 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938653 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.938837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.938740 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:03.940771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.940746 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:18:03.986064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.986013 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"rate-test-model-3d8dc5-kserve-self-signed-certs\"" Nov 03 04:18:03.986415 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:03.986391 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"default-dockercfg-875mt\"" Nov 03 04:18:04.038552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.038520 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"kube-root-ca.crt\"" Nov 03 04:18:04.039863 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.039824 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3d8dc5\"/\"openshift-service-ca.crt\"" Nov 03 04:18:04.041379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.041130 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.041746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.041710 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.042240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.042217 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjftf\" (UniqueName: \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.042423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.042404 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.042618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.042577 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.043160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.043088 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.045041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.043340 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.045041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.043710 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.045041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.044011 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.047407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.047384 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.048012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.047991 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.103327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.103290 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjftf\" (UniqueName: \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\") pod \"rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.202022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.201985 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:04.611790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.611762 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:18:04.612417 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:18:04.612390 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c7d5936_3b42_4299_bef7_050cee937bfb.slice/crio-66979ad81e07c57c610cf76e401e9ca5704874e0b21800863c5401578dd710e6 WatchSource:0}: Error finding container 66979ad81e07c57c610cf76e401e9ca5704874e0b21800863c5401578dd710e6: Status 404 returned error can't find the container with id 66979ad81e07c57c610cf76e401e9ca5704874e0b21800863c5401578dd710e6 Nov 03 04:18:04.753361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.753236 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 04:18:04.753361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:04.753299 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 04:18:05.042384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.042342 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" event={"ID":"3c7d5936-3b42-4299-bef7-050cee937bfb","Type":"ContainerStarted","Data":"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25"} Nov 03 04:18:05.042793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.042399 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" event={"ID":"3c7d5936-3b42-4299-bef7-050cee937bfb","Type":"ContainerStarted","Data":"66979ad81e07c57c610cf76e401e9ca5704874e0b21800863c5401578dd710e6"} Nov 03 04:18:05.169870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.160706 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" podStartSLOduration=2.160686858 podStartE2EDuration="2.160686858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:18:05.160249282 +0000 UTC m=+4576.312937703" watchObservedRunningTime="2026-11-03 04:18:05.160686858 +0000 UTC m=+4576.313375280" Nov 03 04:18:05.202170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.202134 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:05.751332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.751288 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 04:18:05.751657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:05.751625 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 04:18:06.755514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:06.755478 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:18:06.756021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:06.755989 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:18:06.756703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:06.756678 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:18:11.072455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.072406 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4"] Nov 03 04:18:11.075729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.075703 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.195869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.182555 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:18:11.195869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.182679 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:18:11.195869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.182746 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:18:11.195869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.182813 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:18:11.258383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.258300 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4"] Nov 03 04:18:11.282877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.282807 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.283107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.283008 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klckq\" (UniqueName: \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.283204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.283167 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.384748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.384651 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klckq\" (UniqueName: \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.384995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.384758 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.384995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.384837 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.385519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.385499 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.387597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.387572 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.445979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.445948 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klckq\" (UniqueName: \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\") pod \"maas-api-revoke-sub-0b804f98-mlgb4\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:11.689609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:11.689515 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:12.076674 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:12.076643 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4"] Nov 03 04:18:12.078521 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:18:12.078488 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2b5472_a3d6_4842_8c26_2ed87140bdff.slice/crio-6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d WatchSource:0}: Error finding container 6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d: Status 404 returned error can't find the container with id 6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d Nov 03 04:18:12.115421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:12.115377 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" event={"ID":"8d2b5472-a3d6-4842-8c26-2ed87140bdff","Type":"ContainerStarted","Data":"6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d"} Nov 03 04:18:13.124147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:13.124105 5275 generic.go:356] "Generic (PLEG): container finished" podID="8d2b5472-a3d6-4842-8c26-2ed87140bdff" containerID="b8a237660d4ecccf01374b82fe32d24977b0662e658d1aa65c1d5cf02721aa88" exitCode=0 Nov 03 04:18:13.124539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:13.124185 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" event={"ID":"8d2b5472-a3d6-4842-8c26-2ed87140bdff","Type":"ContainerDied","Data":"b8a237660d4ecccf01374b82fe32d24977b0662e658d1aa65c1d5cf02721aa88"} Nov 03 04:18:14.206775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.206742 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:14.224958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.224929 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:14.325952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.325919 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:14.429820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.429733 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\") pod \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " Nov 03 04:18:14.430025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.429882 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\" (UniqueName: \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\") pod \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " Nov 03 04:18:14.430025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.429928 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\") pod \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\" (UID: \"8d2b5472-a3d6-4842-8c26-2ed87140bdff\") " Nov 03 04:18:14.430185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.430156 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca" pod "8d2b5472-a3d6-4842-8c26-2ed87140bdff" (UID: "8d2b5472-a3d6-4842-8c26-2ed87140bdff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:18:14.430380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.430362 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:18:14.432387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.432348 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca" pod "8d2b5472-a3d6-4842-8c26-2ed87140bdff" (UID: "8d2b5472-a3d6-4842-8c26-2ed87140bdff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:18:14.432514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.432448 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq" pod "8d2b5472-a3d6-4842-8c26-2ed87140bdff" (UID: "8d2b5472-a3d6-4842-8c26-2ed87140bdff"). InnerVolumeSpecName "kube-api-access-klckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:18:14.531545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.531500 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klckq\" (UniqueName: \"kubernetes.io/projected/8d2b5472-a3d6-4842-8c26-2ed87140bdff-kube-api-access-klckq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:18:14.531545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:14.531538 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2b5472-a3d6-4842-8c26-2ed87140bdff-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:18:15.140118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:15.140087 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" Nov 03 04:18:15.140278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:15.140118 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4" event={"ID":"8d2b5472-a3d6-4842-8c26-2ed87140bdff","Type":"ContainerDied","Data":"6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d"} Nov 03 04:18:15.140278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:15.140169 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bfb77b6f6f66f48502d16d3b6f860f8b0df5df618322aa1cedb89f2089c8a1d" Nov 03 04:18:15.152185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:15.152153 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:18:32.330183 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:18:32.330106 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:18:43.629307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.629260 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:18:43.630070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.630051 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8d2b5472-a3d6-4842-8c26-2ed87140bdff" containerName="revoke-keys" Nov 03 04:18:43.630070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.630070 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2b5472-a3d6-4842-8c26-2ed87140bdff" containerName="revoke-keys" Nov 03 04:18:43.630189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.630180 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2b5472-a3d6-4842-8c26-2ed87140bdff" containerName="revoke-keys" Nov 03 04:18:43.634273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.634251 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.686713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.686671 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-68b670-openshift-default-dockercfg-tnxvt\"" Nov 03 04:18:43.708828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.708786 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.708828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.708829 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.708956 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709069 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709135 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvsj7\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709242 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709337 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709365 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.709426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.709398 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.740529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.740496 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:18:43.811378 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811333 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811423 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvsj7\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811462 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811521 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811545 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811665 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811827 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811931 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.811993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.811970 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.812166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.812094 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.812166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.812128 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.812260 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.812187 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.812369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.812338 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.812732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.812712 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.814742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.814715 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.815067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.815043 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.872758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.872724 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.872991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.872918 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvsj7\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:43.947996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:43.947914 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:44.396535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:44.396490 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:18:44.406966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:44.406913 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:18:44.407090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:44.407068 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:18:44.407179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:44.407152 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:18:45.412098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.412057 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" event={"ID":"37ab262c-0e7a-4066-acaf-1af032676c90","Type":"ContainerStarted","Data":"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac"} Nov 03 04:18:45.412536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.412123 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" event={"ID":"37ab262c-0e7a-4066-acaf-1af032676c90","Type":"ContainerStarted","Data":"392b3c6a2dc37af7a0cb191608225e4c081ee727cabc7e30a50ed6e6a2e1fa3f"} Nov 03 04:18:45.583643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.583573 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podStartSLOduration=2.583551718 podStartE2EDuration="2.583551718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:18:45.582446105 +0000 UTC m=+4616.735134527" watchObservedRunningTime="2026-11-03 04:18:45.583551718 +0000 UTC m=+4616.736240140" Nov 03 04:18:45.948158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.948115 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:45.954871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.954019 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body= Nov 03 04:18:45.954871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:45.954082 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" Nov 03 04:18:46.952874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:46.949209 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body= Nov 03 04:18:46.952874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:46.949273 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" Nov 03 04:18:47.948179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:47.948144 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body= Nov 03 04:18:47.948393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:47.948190 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" Nov 03 04:18:48.952876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:48.949027 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body= Nov 03 04:18:48.952876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:48.949085 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" Nov 03 04:18:49.954882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:49.949260 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body= Nov 03 04:18:49.954882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:49.949333 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" Nov 03 04:18:50.953562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:50.953523 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:50.953879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:50.953830 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:50.954550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:50.954535 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:18:54.049628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:54.049598 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5"] Nov 03 04:18:54.098914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:54.098882 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba70333-qwkq5"] Nov 03 04:18:54.196067 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:54.196037 5275 scope.go:122] "RemoveContainer" containerID="cb76596e0794a64e81c485273d4e851b132dc87a308348166c3a05ca252dca83" Nov 03 04:18:54.986795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:18:54.986761 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28614d35-5281-42d3-a4e8-9e743ad301cb" path="/var/lib/kubelet/pods/28614d35-5281-42d3-a4e8-9e743ad301cb/volumes" Nov 03 04:19:00.584391 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:00.584348 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:19:00.695700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:00.695661 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:19:02.345261 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:02.345224 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:19:05.584957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.584909 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" podUID="94e0fa20-3c55-4f37-8805-35b2b011959c" containerName="payload-processing" containerID="cri-o://6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84" gracePeriod=25 Nov 03 04:19:05.696190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.696147 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" podUID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" containerName="payload-pre-processing" containerID="cri-o://c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b" gracePeriod=25 Nov 03 04:19:05.863910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.863743 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:19:05.933494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.933455 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\" (UniqueName: \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\") pod \"94e0fa20-3c55-4f37-8805-35b2b011959c\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " Nov 03 04:19:05.933704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.933504 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\") pod \"94e0fa20-3c55-4f37-8805-35b2b011959c\" (UID: \"94e0fa20-3c55-4f37-8805-35b2b011959c\") " Nov 03 04:19:05.933969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.933943 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume" pod "94e0fa20-3c55-4f37-8805-35b2b011959c" (UID: "94e0fa20-3c55-4f37-8805-35b2b011959c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:05.934127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.934112 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e0fa20-3c55-4f37-8805-35b2b011959c-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:05.937750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.937718 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g" pod "94e0fa20-3c55-4f37-8805-35b2b011959c" (UID: "94e0fa20-3c55-4f37-8805-35b2b011959c"). InnerVolumeSpecName "kube-api-access-qdr4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:05.968473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:05.968446 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:19:06.034817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.034773 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\" (UniqueName: \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\") pod \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " Nov 03 04:19:06.035119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.035097 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\") pod \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\" (UID: \"b980e6e1-3a3b-4074-8a11-70a6eefaab8d\") " Nov 03 04:19:06.035437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.035410 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume" pod "b980e6e1-3a3b-4074-8a11-70a6eefaab8d" (UID: "b980e6e1-3a3b-4074-8a11-70a6eefaab8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:06.035751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.035724 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdr4g\" (UniqueName: \"kubernetes.io/projected/94e0fa20-3c55-4f37-8805-35b2b011959c-kube-api-access-qdr4g\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:06.035839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.035758 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:06.037129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.037109 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj" pod "b980e6e1-3a3b-4074-8a11-70a6eefaab8d" (UID: "b980e6e1-3a3b-4074-8a11-70a6eefaab8d"). InnerVolumeSpecName "kube-api-access-xs9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:06.136924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.136814 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9kj\" (UniqueName: \"kubernetes.io/projected/b980e6e1-3a3b-4074-8a11-70a6eefaab8d-kube-api-access-xs9kj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:06.610154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.610125 5275 generic.go:356] "Generic (PLEG): container finished" podID="94e0fa20-3c55-4f37-8805-35b2b011959c" containerID="6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84" exitCode=0 Nov 03 04:19:06.610601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.610196 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" Nov 03 04:19:06.610601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.610224 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" event={"ID":"94e0fa20-3c55-4f37-8805-35b2b011959c","Type":"ContainerDied","Data":"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84"} Nov 03 04:19:06.610601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.610294 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq" event={"ID":"94e0fa20-3c55-4f37-8805-35b2b011959c","Type":"ContainerDied","Data":"23086bfe7f68a7641fecb34979a6cba9539dd5a24a92744d96697779aa926dc8"} Nov 03 04:19:06.610601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.610326 5275 scope.go:122] "RemoveContainer" containerID="6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84" Nov 03 04:19:06.611921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.611897 5275 generic.go:356] "Generic (PLEG): container finished" podID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" containerID="c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b" exitCode=0 Nov 03 04:19:06.612058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.611952 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" Nov 03 04:19:06.612058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.611950 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" event={"ID":"b980e6e1-3a3b-4074-8a11-70a6eefaab8d","Type":"ContainerDied","Data":"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b"} Nov 03 04:19:06.612172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.612089 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt" event={"ID":"b980e6e1-3a3b-4074-8a11-70a6eefaab8d","Type":"ContainerDied","Data":"a38da8ae3e3fc467826cabebd976555c2f02bb49491d24e2b3f558579e93f53c"} Nov 03 04:19:06.623726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.623700 5275 scope.go:122] "RemoveContainer" containerID="6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84" Nov 03 04:19:06.624123 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:06.624099 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84\": container with ID starting with 6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84 not found: ID does not exist" containerID="6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84" Nov 03 04:19:06.624225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.624130 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84"} err="failed to get container status \"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84\": rpc error: code = NotFound desc = could not find container \"6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84\": container with ID starting with 6bcdc225011e1d154c52bc07c972f0fa4476843755837a10e4b4c6fa4b7e4c84 not found: ID does not exist" Nov 03 04:19:06.624225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.624148 5275 scope.go:122] "RemoveContainer" containerID="c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b" Nov 03 04:19:06.644066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.644038 5275 scope.go:122] "RemoveContainer" containerID="c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b" Nov 03 04:19:06.645393 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:06.645364 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b\": container with ID starting with c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b not found: ID does not exist" containerID="c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b" Nov 03 04:19:06.645501 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.645404 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b"} err="failed to get container status \"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b\": rpc error: code = NotFound desc = could not find container \"c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b\": container with ID starting with c8570952661f9c0102f4224b40f3a918428b8c51dad293cc9b42aa244218894b not found: ID does not exist" Nov 03 04:19:06.777282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.777243 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:19:06.827288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.827250 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-255c49-764c6c6b9-mjthq"] Nov 03 04:19:06.890906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.890792 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:19:06.894781 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.894752 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-255c49-77bd6cbb99-rvsmt"] Nov 03 04:19:06.988513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.988476 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e0fa20-3c55-4f37-8805-35b2b011959c" path="/var/lib/kubelet/pods/94e0fa20-3c55-4f37-8805-35b2b011959c/volumes" Nov 03 04:19:06.989179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:06.989157 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" path="/var/lib/kubelet/pods/b980e6e1-3a3b-4074-8a11-70a6eefaab8d/volumes" Nov 03 04:19:20.103078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:20.103036 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:19:20.103632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:20.103307 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" containerID="cri-o://805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac" gracePeriod=30 Nov 03 04:19:20.956397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:20.956361 5275 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:19:20.956584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:20.956418 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:19:24.339913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:24.339871 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:19:24.449474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:24.449433 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:19:25.393677 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.393634 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:19:25.403996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.403957 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404050 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404084 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404140 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404210 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404261 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404542 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404389 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404542 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404466 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\") pod \"37ab262c-0e7a-4066-acaf-1af032676c90\" (UID: \"37ab262c-0e7a-4066-acaf-1af032676c90\") " Nov 03 04:19:25.404542 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404492 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.404713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404543 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.404713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404567 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:25.404834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404775 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.404970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.404941 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.405108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.405086 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.405108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.405110 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.405272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.405124 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.405272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.405137 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ab262c-0e7a-4066-acaf-1af032676c90-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.405272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.405150 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.406696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.406673 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "kube-api-access-fvsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:25.406886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.406836 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:25.407186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.407166 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.407243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.407184 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo" pod "37ab262c-0e7a-4066-acaf-1af032676c90" (UID: "37ab262c-0e7a-4066-acaf-1af032676c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:19:25.506787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.506751 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvsj7\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-kube-api-access-fvsj7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.506787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.506783 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ab262c-0e7a-4066-acaf-1af032676c90-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.506787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.506794 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ab262c-0e7a-4066-acaf-1af032676c90-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.507077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.506803 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ab262c-0e7a-4066-acaf-1af032676c90-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.815149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.813946 5275 generic.go:356] "Generic (PLEG): container finished" podID="37ab262c-0e7a-4066-acaf-1af032676c90" containerID="805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac" exitCode=0 Nov 03 04:19:25.815149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.814347 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" Nov 03 04:19:25.815149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.814472 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" event={"ID":"37ab262c-0e7a-4066-acaf-1af032676c90","Type":"ContainerDied","Data":"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac"} Nov 03 04:19:25.815149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.814817 5275 scope.go:122] "RemoveContainer" containerID="805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac" Nov 03 04:19:25.819868 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.816927 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn" event={"ID":"37ab262c-0e7a-4066-acaf-1af032676c90","Type":"ContainerDied","Data":"392b3c6a2dc37af7a0cb191608225e4c081ee727cabc7e30a50ed6e6a2e1fa3f"} Nov 03 04:19:25.838203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.837871 5275 scope.go:122] "RemoveContainer" containerID="805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac" Nov 03 04:19:25.838586 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:25.838556 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac\": container with ID starting with 805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac not found: ID does not exist" containerID="805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac" Nov 03 04:19:25.838716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.838597 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac"} err="failed to get container status \"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac\": rpc error: code = NotFound desc = could not find container \"805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac\": container with ID starting with 805c2d175fd1476f8ff6f90ea7c6ce6693d8c3f7085682121064b58918a3bcac not found: ID does not exist" Nov 03 04:19:25.985056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:25.985023 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:19:26.030637 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:26.030601 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-tb7xn"] Nov 03 04:19:26.985682 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:26.985646 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" path="/var/lib/kubelet/pods/37ab262c-0e7a-4066-acaf-1af032676c90/volumes" Nov 03 04:19:29.340574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.340530 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" podUID="935196d0-bd74-4d5d-addd-99261d675799" containerName="payload-processing" containerID="cri-o://db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371" gracePeriod=25 Nov 03 04:19:29.450694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.450648 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" podUID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" containerName="payload-pre-processing" containerID="cri-o://98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5" gracePeriod=25 Nov 03 04:19:29.632860 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.632819 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:19:29.661096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.661057 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\" (UniqueName: \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\") pod \"935196d0-bd74-4d5d-addd-99261d675799\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " Nov 03 04:19:29.661312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.661293 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\") pod \"935196d0-bd74-4d5d-addd-99261d675799\" (UID: \"935196d0-bd74-4d5d-addd-99261d675799\") " Nov 03 04:19:29.662485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.662456 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume" pod "935196d0-bd74-4d5d-addd-99261d675799" (UID: "935196d0-bd74-4d5d-addd-99261d675799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:29.666309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.666277 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq" pod "935196d0-bd74-4d5d-addd-99261d675799" (UID: "935196d0-bd74-4d5d-addd-99261d675799"). InnerVolumeSpecName "kube-api-access-7w7lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:29.731332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.731306 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:19:29.763243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763204 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\" (UniqueName: \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\") pod \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " Nov 03 04:19:29.763423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763372 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\") pod \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\" (UID: \"fbbc470b-839c-4a3d-a1c0-c1679e0aab78\") " Nov 03 04:19:29.763738 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763710 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume" pod "fbbc470b-839c-4a3d-a1c0-c1679e0aab78" (UID: "fbbc470b-839c-4a3d-a1c0-c1679e0aab78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:29.763982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763919 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935196d0-bd74-4d5d-addd-99261d675799-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:29.763982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763945 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7lq\" (UniqueName: \"kubernetes.io/projected/935196d0-bd74-4d5d-addd-99261d675799-kube-api-access-7w7lq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:29.763982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.763961 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:29.765680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.765658 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms" pod "fbbc470b-839c-4a3d-a1c0-c1679e0aab78" (UID: "fbbc470b-839c-4a3d-a1c0-c1679e0aab78"). InnerVolumeSpecName "kube-api-access-hwfms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:29.847173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.847137 5275 generic.go:356] "Generic (PLEG): container finished" podID="935196d0-bd74-4d5d-addd-99261d675799" containerID="db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371" exitCode=0 Nov 03 04:19:29.847388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.847183 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" Nov 03 04:19:29.847388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.847230 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" event={"ID":"935196d0-bd74-4d5d-addd-99261d675799","Type":"ContainerDied","Data":"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371"} Nov 03 04:19:29.847388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.847282 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2" event={"ID":"935196d0-bd74-4d5d-addd-99261d675799","Type":"ContainerDied","Data":"66bdb14df407602b7395c6d5c2830be66b6f81e70f7bf10ae98006ec4c706b6b"} Nov 03 04:19:29.847388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.847307 5275 scope.go:122] "RemoveContainer" containerID="db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371" Nov 03 04:19:29.848937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.848918 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" Nov 03 04:19:29.849083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.848967 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" event={"ID":"fbbc470b-839c-4a3d-a1c0-c1679e0aab78","Type":"ContainerDied","Data":"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5"} Nov 03 04:19:29.849083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.848908 5275 generic.go:356] "Generic (PLEG): container finished" podID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" containerID="98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5" exitCode=0 Nov 03 04:19:29.849199 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.849133 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz" event={"ID":"fbbc470b-839c-4a3d-a1c0-c1679e0aab78","Type":"ContainerDied","Data":"2202209bfb91457b1eb244940a6da772a35c368047469f4f764ecbc3ebb9f299"} Nov 03 04:19:29.859168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.859134 5275 scope.go:122] "RemoveContainer" containerID="db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371" Nov 03 04:19:29.859539 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:29.859511 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371\": container with ID starting with db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371 not found: ID does not exist" containerID="db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371" Nov 03 04:19:29.859632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.859541 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371"} err="failed to get container status \"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371\": rpc error: code = NotFound desc = could not find container \"db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371\": container with ID starting with db60df30b06ae830d503808cedfeabd13e9fad5bb12478f1ecce2c653650f371 not found: ID does not exist" Nov 03 04:19:29.859632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.859560 5275 scope.go:122] "RemoveContainer" containerID="98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5" Nov 03 04:19:29.865126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.865097 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfms\" (UniqueName: \"kubernetes.io/projected/fbbc470b-839c-4a3d-a1c0-c1679e0aab78-kube-api-access-hwfms\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:29.871760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.871737 5275 scope.go:122] "RemoveContainer" containerID="98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5" Nov 03 04:19:29.872168 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:29.872139 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5\": container with ID starting with 98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5 not found: ID does not exist" containerID="98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5" Nov 03 04:19:29.872248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.872179 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5"} err="failed to get container status \"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5\": rpc error: code = NotFound desc = could not find container \"98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5\": container with ID starting with 98b02347c6880a639f7159b810cc01be4730af7d5c6d5139e11e532cf44c06b5 not found: ID does not exist" Nov 03 04:19:29.975513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.975465 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:19:29.978925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:29.978893 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30496c2-596bdd8c67-wd4fz"] Nov 03 04:19:30.087585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:30.087523 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:19:30.091424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:30.091393 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30496c2-9f9cc4df5-hgcm2"] Nov 03 04:19:30.985279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:30.985245 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935196d0-bd74-4d5d-addd-99261d675799" path="/var/lib/kubelet/pods/935196d0-bd74-4d5d-addd-99261d675799/volumes" Nov 03 04:19:30.985689 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:30.985654 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" path="/var/lib/kubelet/pods/fbbc470b-839c-4a3d-a1c0-c1679e0aab78/volumes" Nov 03 04:19:32.329764 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:19:32.329721 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:19:54.241957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:54.241877 5275 scope.go:122] "RemoveContainer" containerID="2a3b16c1e62519b57cad9c37cb2744c685ce6d60b8d802468708beb06157b657" Nov 03 04:19:54.253523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:54.253494 5275 scope.go:122] "RemoveContainer" containerID="de967cdb3a5d9aee83818f2cb34cd98715e72921359633a8c89861c9c8d0696b" Nov 03 04:19:54.264317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:19:54.264293 5275 scope.go:122] "RemoveContainer" containerID="b76acd3c04a93d28303391dc938e01b87855c485bf2389ac82c48f3fbab72734" Nov 03 04:20:00.846427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:00.846387 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:20:00.846896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:00.846634 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" containerID="cri-o://9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7" gracePeriod=30 Nov 03 04:20:02.330789 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:02.330754 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:20:06.125941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.125872 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:20:06.135267 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.135232 5275 generic.go:356] "Generic (PLEG): container finished" podID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerID="9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7" exitCode=0 Nov 03 04:20:06.135448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.135295 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" Nov 03 04:20:06.135448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.135315 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerDied","Data":"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7"} Nov 03 04:20:06.135448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.135384 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v" event={"ID":"c3e0c5b8-c155-465a-bf40-16832d2a3ff2","Type":"ContainerDied","Data":"bc7cffd7c17a0229a0f59ad7d6bfdbe4134904cedbf2b7add3f8adae4a18ac9e"} Nov 03 04:20:06.135448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.135415 5275 scope.go:122] "RemoveContainer" containerID="9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7" Nov 03 04:20:06.146269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.146240 5275 scope.go:122] "RemoveContainer" containerID="79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6" Nov 03 04:20:06.157406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.157382 5275 scope.go:122] "RemoveContainer" containerID="9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7" Nov 03 04:20:06.157748 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:06.157727 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7\": container with ID starting with 9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7 not found: ID does not exist" containerID="9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7" Nov 03 04:20:06.157862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.157755 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7"} err="failed to get container status \"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7\": rpc error: code = NotFound desc = could not find container \"9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7\": container with ID starting with 9ceff21473af925cb08dff1709bc761d75b8fc013054dba464641baa56fb4fc7 not found: ID does not exist" Nov 03 04:20:06.157862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.157774 5275 scope.go:122] "RemoveContainer" containerID="79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6" Nov 03 04:20:06.158075 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:06.158055 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6\": container with ID starting with 79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6 not found: ID does not exist" containerID="79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6" Nov 03 04:20:06.158138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.158082 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6"} err="failed to get container status \"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6\": rpc error: code = NotFound desc = could not find container \"79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6\": container with ID starting with 79b6f669cd27322107ed650589eabf0465cbf9f0f8e7db07913752560880e5b6 not found: ID does not exist" Nov 03 04:20:06.256633 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256596 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.256817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256711 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.256817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256757 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.256817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256785 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.256817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256812 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.257109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256904 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.257109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.256970 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.257109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257043 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.257282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257112 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\") pod \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\" (UID: \"c3e0c5b8-c155-465a-bf40-16832d2a3ff2\") " Nov 03 04:20:06.257282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257128 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:06.257282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257198 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:20:06.257449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257311 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:06.257535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257503 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:06.257758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257729 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.257889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257763 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.257889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257778 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.257889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257793 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.258068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.257898 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:06.259938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.259909 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:06.260077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.259954 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "kube-api-access-dlvml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:06.260077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.260031 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:20:06.260216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.260187 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy" pod "c3e0c5b8-c155-465a-bf40-16832d2a3ff2" (UID: "c3e0c5b8-c155-465a-bf40-16832d2a3ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:06.359419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.359382 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.359419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.359414 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.359419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.359424 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.359685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.359434 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvml\" (UniqueName: \"kubernetes.io/projected/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-kube-api-access-dlvml\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.359685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.359443 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e0c5b8-c155-465a-bf40-16832d2a3ff2-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:06.560144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.560103 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:20:06.563422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.563387 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-255c49-openshift-default-85bb8b85d7-q2j7v"] Nov 03 04:20:06.815839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.815802 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:20:06.923858 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.923807 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:20:06.988647 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:06.988607 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" path="/var/lib/kubelet/pods/c3e0c5b8-c155-465a-bf40-16832d2a3ff2/volumes" Nov 03 04:20:11.817160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:11.817108 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" podUID="eafea22a-939c-4781-b06d-1def3a355682" containerName="payload-processing" containerID="cri-o://c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e" gracePeriod=25 Nov 03 04:20:11.924409 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:11.924370 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" podUID="31040ac2-4722-4c7f-80e1-c96772349c66" containerName="payload-pre-processing" containerID="cri-o://49fad223774bb578df683a477a698036146074a137959473c46539772d88104a" gracePeriod=25 Nov 03 04:20:12.117116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.117084 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:20:12.185751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.185715 5275 generic.go:356] "Generic (PLEG): container finished" podID="31040ac2-4722-4c7f-80e1-c96772349c66" containerID="49fad223774bb578df683a477a698036146074a137959473c46539772d88104a" exitCode=0 Nov 03 04:20:12.185992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.185788 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" event={"ID":"31040ac2-4722-4c7f-80e1-c96772349c66","Type":"ContainerDied","Data":"49fad223774bb578df683a477a698036146074a137959473c46539772d88104a"} Nov 03 04:20:12.187720 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.187679 5275 generic.go:356] "Generic (PLEG): container finished" podID="eafea22a-939c-4781-b06d-1def3a355682" containerID="c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e" exitCode=0 Nov 03 04:20:12.187895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.187768 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" Nov 03 04:20:12.187952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.187902 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" event={"ID":"eafea22a-939c-4781-b06d-1def3a355682","Type":"ContainerDied","Data":"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e"} Nov 03 04:20:12.188075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.187951 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2" event={"ID":"eafea22a-939c-4781-b06d-1def3a355682","Type":"ContainerDied","Data":"712c9927b64c55bc8d96981c946d08f032683f0516afaa3395ae3f8c284da03e"} Nov 03 04:20:12.188075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.187982 5275 scope.go:122] "RemoveContainer" containerID="c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e" Nov 03 04:20:12.211262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.211238 5275 scope.go:122] "RemoveContainer" containerID="c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e" Nov 03 04:20:12.211523 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:12.211501 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e\": container with ID starting with c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e not found: ID does not exist" containerID="c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e" Nov 03 04:20:12.211586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.211533 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e"} err="failed to get container status \"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e\": rpc error: code = NotFound desc = could not find container \"c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e\": container with ID starting with c95aa80f78692e188525e73021cbea8186d8bd4aeae394aac40ec2204485a32e not found: ID does not exist" Nov 03 04:20:12.213175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.213157 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:20:12.242205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242171 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\") pod \"eafea22a-939c-4781-b06d-1def3a355682\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " Nov 03 04:20:12.242429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242252 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\") pod \"31040ac2-4722-4c7f-80e1-c96772349c66\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " Nov 03 04:20:12.242429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242337 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\" (UniqueName: \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\") pod \"31040ac2-4722-4c7f-80e1-c96772349c66\" (UID: \"31040ac2-4722-4c7f-80e1-c96772349c66\") " Nov 03 04:20:12.242429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242380 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\" (UniqueName: \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\") pod \"eafea22a-939c-4781-b06d-1def3a355682\" (UID: \"eafea22a-939c-4781-b06d-1def3a355682\") " Nov 03 04:20:12.242713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242649 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume" pod "31040ac2-4722-4c7f-80e1-c96772349c66" (UID: "31040ac2-4722-4c7f-80e1-c96772349c66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:20:12.242713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242650 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume" pod "eafea22a-939c-4781-b06d-1def3a355682" (UID: "eafea22a-939c-4781-b06d-1def3a355682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:20:12.242972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242949 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafea22a-939c-4781-b06d-1def3a355682-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:12.243052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.242977 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31040ac2-4722-4c7f-80e1-c96772349c66-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:12.244987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.244951 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6" pod "eafea22a-939c-4781-b06d-1def3a355682" (UID: "eafea22a-939c-4781-b06d-1def3a355682"). InnerVolumeSpecName "kube-api-access-d25l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:12.244987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.244979 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s" pod "31040ac2-4722-4c7f-80e1-c96772349c66" (UID: "31040ac2-4722-4c7f-80e1-c96772349c66"). InnerVolumeSpecName "kube-api-access-n5g5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:12.344107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.344011 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5g5s\" (UniqueName: \"kubernetes.io/projected/31040ac2-4722-4c7f-80e1-c96772349c66-kube-api-access-n5g5s\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:12.344107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.344045 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d25l6\" (UniqueName: \"kubernetes.io/projected/eafea22a-939c-4781-b06d-1def3a355682-kube-api-access-d25l6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:12.608967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.608866 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:20:12.612320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.612288 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3c1cec-75f7d8dc9-wdkb2"] Nov 03 04:20:12.986487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:12.986395 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafea22a-939c-4781-b06d-1def3a355682" path="/var/lib/kubelet/pods/eafea22a-939c-4781-b06d-1def3a355682/volumes" Nov 03 04:20:13.198430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:13.198387 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" event={"ID":"31040ac2-4722-4c7f-80e1-c96772349c66","Type":"ContainerDied","Data":"715381a901db16acfe91176f141990b1935267a46d15c504176c52c949bb45d8"} Nov 03 04:20:13.198616 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:13.198446 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h" Nov 03 04:20:13.198616 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:13.198518 5275 scope.go:122] "RemoveContainer" containerID="49fad223774bb578df683a477a698036146074a137959473c46539772d88104a" Nov 03 04:20:13.366763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:13.366725 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:20:13.371584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:13.371554 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3c1cec-6d79645f87-8958h"] Nov 03 04:20:14.986011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:14.985974 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31040ac2-4722-4c7f-80e1-c96772349c66" path="/var/lib/kubelet/pods/31040ac2-4722-4c7f-80e1-c96772349c66/volumes" Nov 03 04:20:31.606693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:31.606661 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:20:31.607273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:31.606909 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" containerID="cri-o://a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390" gracePeriod=30 Nov 03 04:20:32.331490 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:32.331453 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:20:36.870949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.870870 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:20:36.973479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973439 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.973479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973486 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.973748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973516 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.973748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973562 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.973748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973607 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.973748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973651 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.974016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973788 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:36.974016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973883 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.974016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973937 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.974016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973965 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\") pod \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\" (UID: \"d544c042-ccbf-4de9-bce4-50bdcd8c4de1\") " Nov 03 04:20:36.974016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.973963 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:36.974320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974194 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:36.974320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974203 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:20:36.974438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974387 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:36.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974698 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:36.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974730 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:36.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974746 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:36.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974760 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:36.974771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.974775 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:36.976210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.976183 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:20:36.976331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.976290 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:20:36.976552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.976529 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:36.976552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:36.976545 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742" pod "d544c042-ccbf-4de9-bce4-50bdcd8c4de1" (UID: "d544c042-ccbf-4de9-bce4-50bdcd8c4de1"). InnerVolumeSpecName "kube-api-access-rw742". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:37.075540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.075506 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:37.075540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.075539 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:37.075761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.075550 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw742\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-kube-api-access-rw742\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:37.075761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.075561 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d544c042-ccbf-4de9-bce4-50bdcd8c4de1-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:37.378742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.378705 5275 generic.go:356] "Generic (PLEG): container finished" podID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerID="a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390" exitCode=0 Nov 03 04:20:37.378981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.378763 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" Nov 03 04:20:37.378981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.378793 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerDied","Data":"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390"} Nov 03 04:20:37.378981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.378892 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb" event={"ID":"d544c042-ccbf-4de9-bce4-50bdcd8c4de1","Type":"ContainerDied","Data":"13e1382eaf63f34a81febf1a869dfcc33673608aeeb8aa87462f72364415d2ed"} Nov 03 04:20:37.378981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.378926 5275 scope.go:122] "RemoveContainer" containerID="a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390" Nov 03 04:20:37.389434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.389414 5275 scope.go:122] "RemoveContainer" containerID="78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50" Nov 03 04:20:37.399230 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.399207 5275 scope.go:122] "RemoveContainer" containerID="a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390" Nov 03 04:20:37.399535 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:37.399518 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390\": container with ID starting with a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390 not found: ID does not exist" containerID="a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390" Nov 03 04:20:37.399618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.399549 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390"} err="failed to get container status \"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390\": rpc error: code = NotFound desc = could not find container \"a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390\": container with ID starting with a2c68df157979871ab8b877953bacef6b34e0224b90a558baddb637de1ad1390 not found: ID does not exist" Nov 03 04:20:37.399618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.399574 5275 scope.go:122] "RemoveContainer" containerID="78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50" Nov 03 04:20:37.399882 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:20:37.399834 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50\": container with ID starting with 78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50 not found: ID does not exist" containerID="78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50" Nov 03 04:20:37.399948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.399890 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50"} err="failed to get container status \"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50\": rpc error: code = NotFound desc = could not find container \"78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50\": container with ID starting with 78cd9cfbe924b2337cccf0d96fe0353c74f5a9b05d9aa8f52acc5a95d9f47d50 not found: ID does not exist" Nov 03 04:20:37.543833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.543791 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:20:37.595150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:37.595109 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e30496c2-openshift-default-99bb67bdf-qxrmb"] Nov 03 04:20:38.985291 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:38.985258 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" path="/var/lib/kubelet/pods/d544c042-ccbf-4de9-bce4-50bdcd8c4de1/volumes" Nov 03 04:20:46.746396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:46.746356 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 04:20:51.747655 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:51.747612 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" containerID="cri-o://82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525" gracePeriod=25 Nov 03 04:20:52.724466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:20:52.724420 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:02.331022 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:02.330979 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:21:02.724113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:02.724028 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:06.407468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:06.407418 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 04:21:06.654418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:06.654373 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 04:21:11.408977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:11.408903 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" containerID="cri-o://c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320" gracePeriod=25 Nov 03 04:21:11.655316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:11.655272 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" containerID="cri-o://df9a8762180db24809f0bd9f7c56e3a5ce4da524804e91662866707d076ac18e" gracePeriod=25 Nov 03 04:21:12.022999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.022970 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 04:21:12.136922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.136874 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\" (UniqueName: \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\") pod \"58b4a33a-3c01-4549-b9b9-a1178b430780\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " Nov 03 04:21:12.137130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.136970 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\") pod \"58b4a33a-3c01-4549-b9b9-a1178b430780\" (UID: \"58b4a33a-3c01-4549-b9b9-a1178b430780\") " Nov 03 04:21:12.137423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.137392 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume" pod "58b4a33a-3c01-4549-b9b9-a1178b430780" (UID: "58b4a33a-3c01-4549-b9b9-a1178b430780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:12.139469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.139444 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d" pod "58b4a33a-3c01-4549-b9b9-a1178b430780" (UID: "58b4a33a-3c01-4549-b9b9-a1178b430780"). InnerVolumeSpecName "kube-api-access-s6x6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:12.238148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.238108 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6x6d\" (UniqueName: \"kubernetes.io/projected/58b4a33a-3c01-4549-b9b9-a1178b430780-kube-api-access-s6x6d\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:12.238346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.238174 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b4a33a-3c01-4549-b9b9-a1178b430780-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:12.621896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.621858 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v"] Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622423 5275 cpu_manager.go:414] "containerMap: removing container" podUID="935196d0-bd74-4d5d-addd-99261d675799" containerName="payload-processing" Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622439 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935196d0-bd74-4d5d-addd-99261d675799" containerName="payload-processing" Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622464 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622468 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622477 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622482 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622492 5275 cpu_manager.go:414] "containerMap: removing container" podUID="94e0fa20-3c55-4f37-8805-35b2b011959c" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622497 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e0fa20-3c55-4f37-8805-35b2b011959c" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622504 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622509 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622515 5275 cpu_manager.go:414] "containerMap: removing container" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622520 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622525 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622529 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622546 5275 cpu_manager.go:414] "containerMap: removing container" podUID="eafea22a-939c-4781-b06d-1def3a355682" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622550 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafea22a-939c-4781-b06d-1def3a355682" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622559 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622563 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622576 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622580 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622587 5275 cpu_manager.go:414] "containerMap: removing container" podUID="31040ac2-4722-4c7f-80e1-c96772349c66" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622591 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31040ac2-4722-4c7f-80e1-c96772349c66" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622601 5275 cpu_manager.go:414] "containerMap: removing container" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622605 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622680 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622689 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622696 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e0fa20-3c55-4f37-8805-35b2b011959c" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622702 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d544c042-ccbf-4de9-bce4-50bdcd8c4de1" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622709 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622715 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e0c5b8-c155-465a-bf40-16832d2a3ff2" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622722 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafea22a-939c-4781-b06d-1def3a355682" containerName="payload-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622728 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbc470b-839c-4a3d-a1c0-c1679e0aab78" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622735 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b980e6e1-3a3b-4074-8a11-70a6eefaab8d" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622741 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="31040ac2-4722-4c7f-80e1-c96772349c66" containerName="payload-pre-processing" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622748 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ab262c-0e7a-4066-acaf-1af032676c90" containerName="istio-proxy" Nov 03 04:21:12.622817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.622755 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="935196d0-bd74-4d5d-addd-99261d675799" containerName="payload-processing" Nov 03 04:21:12.626514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.626496 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.640548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.640520 5275 generic.go:356] "Generic (PLEG): container finished" podID="58b4a33a-3c01-4549-b9b9-a1178b430780" containerID="82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525" exitCode=0 Nov 03 04:21:12.640729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.640563 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" event={"ID":"58b4a33a-3c01-4549-b9b9-a1178b430780","Type":"ContainerDied","Data":"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525"} Nov 03 04:21:12.640729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.640580 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" Nov 03 04:21:12.640729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.640602 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d" event={"ID":"58b4a33a-3c01-4549-b9b9-a1178b430780","Type":"ContainerDied","Data":"0884febff198a7de22b5fe4c46c02fab7285709e6e41c7fc340cb070bb0f72c1"} Nov 03 04:21:12.640729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.640625 5275 scope.go:122] "RemoveContainer" containerID="82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525" Nov 03 04:21:12.642423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.642398 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.642597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.642577 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.642725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.642707 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshvc\" (UniqueName: \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.651512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.651494 5275 scope.go:122] "RemoveContainer" containerID="82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525" Nov 03 04:21:12.651892 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:12.651836 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525\": container with ID starting with 82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525 not found: ID does not exist" containerID="82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525" Nov 03 04:21:12.652007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.651900 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525"} err="failed to get container status \"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525\": rpc error: code = NotFound desc = could not find container \"82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525\": container with ID starting with 82685779e4dde08949a55ade4cf92e6acacbe4e0c9a187ff8b539cf8ac1c0525 not found: ID does not exist" Nov 03 04:21:12.679210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.679180 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:21:12.679973 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.679215 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:21:12.683272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.683246 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v"] Nov 03 04:21:12.721336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.721300 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:21:12.721545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.721365 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:21:12.743557 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.743515 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.743772 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.743649 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.743772 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.743743 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshvc\" (UniqueName: \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.744431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.744348 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.746241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.746216 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.807073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.807035 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshvc\" (UniqueName: \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\") pod \"maas-api-revoke-sub-076d139b-gps5v\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.858374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.858334 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 04:21:12.863078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.863044 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-10120d-b4c8b494c-zkm5d"] Nov 03 04:21:12.941758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.941675 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:12.986507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:12.986472 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b4a33a-3c01-4549-b9b9-a1178b430780" path="/var/lib/kubelet/pods/58b4a33a-3c01-4549-b9b9-a1178b430780/volumes" Nov 03 04:21:13.324676 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:21:13.324643 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76aa0d8f_2157_4f7c_8f5c_058bf112bf3d.slice/crio-ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c WatchSource:0}: Error finding container ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c: Status 404 returned error can't find the container with id ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c Nov 03 04:21:13.324838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:13.324698 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v"] Nov 03 04:21:13.648722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:13.648686 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" event={"ID":"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d","Type":"ContainerStarted","Data":"ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c"} Nov 03 04:21:14.609015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:14.608974 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:14.609235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:14.609010 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:14.666556 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:14.666518 5275 generic.go:356] "Generic (PLEG): container finished" podID="76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" containerID="f14f1bd3801dde331223f1548612191bf588cb4de25c2035820a359ae131f7d8" exitCode=0 Nov 03 04:21:14.667050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:14.666552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" event={"ID":"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d","Type":"ContainerDied","Data":"f14f1bd3801dde331223f1548612191bf588cb4de25c2035820a359ae131f7d8"} Nov 03 04:21:15.845242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.845213 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:15.907327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.907289 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\") pod \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " Nov 03 04:21:15.907625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.907603 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\" (UniqueName: \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\") pod \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " Nov 03 04:21:15.907757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.907741 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\") pod \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\" (UID: \"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d\") " Nov 03 04:21:15.908249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.908223 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca" pod "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" (UID: "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:15.909626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.909595 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:15.915076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.915050 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc" pod "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" (UID: "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d"). InnerVolumeSpecName "kube-api-access-xshvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:15.923456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:15.923406 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca" pod "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" (UID: "76aa0d8f-2157-4f7c-8f5c-058bf112bf3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:21:16.010975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:16.010920 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:16.010975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:16.010952 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshvc\" (UniqueName: \"kubernetes.io/projected/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d-kube-api-access-xshvc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:16.682984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:16.682954 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" Nov 03 04:21:16.682984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:16.682971 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v" event={"ID":"76aa0d8f-2157-4f7c-8f5c-058bf112bf3d","Type":"ContainerDied","Data":"ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c"} Nov 03 04:21:16.683224 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:16.683016 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae521c662bec60aa0ce07d72caccc6279c8a6cd9578a3c5f1bce1d3dd6957b1c" Nov 03 04:21:20.538383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:20.538350 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:21:20.538829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:20.538607 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" containerID="cri-o://3de9939faf79c3affc37943b204635ed8c8ab8bf4c1ffeec4b19d7a5570985ce" gracePeriod=30 Nov 03 04:21:21.758377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:21.758343 5275 patch_prober.go:29] interesting pod/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:21:21.758809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:21.758404 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:21:22.707874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:22.707771 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 04:21:22.708114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:22.708057 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" containerID="cri-o://35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b" gracePeriod=30 Nov 03 04:21:23.326019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.325984 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p"] Nov 03 04:21:23.326570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.326556 5275 cpu_manager.go:414] "containerMap: removing container" podUID="76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" containerName="revoke-keys" Nov 03 04:21:23.326622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.326570 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" containerName="revoke-keys" Nov 03 04:21:23.326660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.326656 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" containerName="revoke-keys" Nov 03 04:21:23.330060 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.330043 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.383573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.383539 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:21:23.383820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.383590 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:21:23.383937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.383662 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:21:23.384632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.384125 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:21:23.387729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.387696 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p"] Nov 03 04:21:23.410293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.410254 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.410550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.410524 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.410637 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.410597 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prsf\" (UniqueName: \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.512113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.512074 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.512309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.512140 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prsf\" (UniqueName: \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.512309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.512240 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.512809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.512785 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.514723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.514700 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.573537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.573502 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prsf\" (UniqueName: \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\") pod \"maas-api-revoke-sub-e0144e8b-txq5p\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:23.644338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:23.644250 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:24.029689 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.029657 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p"] Nov 03 04:21:24.031096 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:21:24.031071 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc30d173_29dd_4c73_95b5_78046c770fa5.slice/crio-48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9 WatchSource:0}: Error finding container 48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9: Status 404 returned error can't find the container with id 48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9 Nov 03 04:21:24.609776 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.609682 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:24.610278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.609675 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:24.744115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.744071 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" event={"ID":"cc30d173-29dd-4c73-95b5-78046c770fa5","Type":"ContainerStarted","Data":"a0dbf8624f2f1ba4aac74f6ab0ed0436c3d0dea66d304995ac19609dfe9ed7d3"} Nov 03 04:21:24.744332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.744164 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" event={"ID":"cc30d173-29dd-4c73-95b5-78046c770fa5","Type":"ContainerStarted","Data":"48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9"} Nov 03 04:21:24.862101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:24.861994 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" podStartSLOduration=1.861976999 podStartE2EDuration="1.861976999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:21:24.858434385 +0000 UTC m=+4776.011122808" watchObservedRunningTime="2026-11-03 04:21:24.861976999 +0000 UTC m=+4776.014665424" Nov 03 04:21:25.756631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.756591 5275 generic.go:356] "Generic (PLEG): container finished" podID="cc30d173-29dd-4c73-95b5-78046c770fa5" containerID="a0dbf8624f2f1ba4aac74f6ab0ed0436c3d0dea66d304995ac19609dfe9ed7d3" exitCode=0 Nov 03 04:21:25.757147 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.756730 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" event={"ID":"cc30d173-29dd-4c73-95b5-78046c770fa5","Type":"ContainerDied","Data":"a0dbf8624f2f1ba4aac74f6ab0ed0436c3d0dea66d304995ac19609dfe9ed7d3"} Nov 03 04:21:25.759686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.759656 5275 generic.go:356] "Generic (PLEG): container finished" podID="7cffde05-c4db-4390-b877-2bca581764da" containerID="3de9939faf79c3affc37943b204635ed8c8ab8bf4c1ffeec4b19d7a5570985ce" exitCode=0 Nov 03 04:21:25.759890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.759720 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerDied","Data":"3de9939faf79c3affc37943b204635ed8c8ab8bf4c1ffeec4b19d7a5570985ce"} Nov 03 04:21:25.759890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.759779 5275 scope.go:122] "RemoveContainer" containerID="649d7460bd0c49ada3e20bb0942d3b422b347823e830b7989e698614f3ef066e" Nov 03 04:21:25.813959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.813879 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:21:25.939127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939087 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939145 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939301 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939356 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939556 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939604 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939629 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:25.939779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939646 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.939779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939674 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939732 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939806 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939870 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939747 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\") pod \"7cffde05-c4db-4390-b877-2bca581764da\" (UID: \"7cffde05-c4db-4390-b877-2bca581764da\") " Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.939980 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.940356 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.940376 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.940389 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.940402 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cffde05-c4db-4390-b877-2bca581764da-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:25.940587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.940415 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:25.941616 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.941595 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:25.941712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.941682 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:21:25.942136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.942117 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "kube-api-access-tl28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:25.942233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:25.942143 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy" pod "7cffde05-c4db-4390-b877-2bca581764da" (UID: "7cffde05-c4db-4390-b877-2bca581764da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:26.041436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.041401 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cffde05-c4db-4390-b877-2bca581764da-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:26.041436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.041435 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl28t\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-kube-api-access-tl28t\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:26.041651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.041449 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cffde05-c4db-4390-b877-2bca581764da-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:26.041651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.041464 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cffde05-c4db-4390-b877-2bca581764da-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:26.769519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.769482 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" Nov 03 04:21:26.769519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.769506 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z" event={"ID":"7cffde05-c4db-4390-b877-2bca581764da","Type":"ContainerDied","Data":"7b9f70281ea7c20a07393905eb0b02e665df273e0725fc8dbec5a31dc8845364"} Nov 03 04:21:26.770109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.769577 5275 scope.go:122] "RemoveContainer" containerID="3de9939faf79c3affc37943b204635ed8c8ab8bf4c1ffeec4b19d7a5570985ce" Nov 03 04:21:26.938621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:26.938580 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:27.004572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.004535 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:21:27.052536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.052498 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\" (UniqueName: \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\") pod \"cc30d173-29dd-4c73-95b5-78046c770fa5\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " Nov 03 04:21:27.052727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.052641 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\") pod \"cc30d173-29dd-4c73-95b5-78046c770fa5\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " Nov 03 04:21:27.052808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.052760 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\") pod \"cc30d173-29dd-4c73-95b5-78046c770fa5\" (UID: \"cc30d173-29dd-4c73-95b5-78046c770fa5\") " Nov 03 04:21:27.055892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.053144 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3c1cec-openshift-default-d4cfcfd57-f7f9z"] Nov 03 04:21:27.055892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.053293 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca" pod "cc30d173-29dd-4c73-95b5-78046c770fa5" (UID: "cc30d173-29dd-4c73-95b5-78046c770fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:27.056894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.056863 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf" pod "cc30d173-29dd-4c73-95b5-78046c770fa5" (UID: "cc30d173-29dd-4c73-95b5-78046c770fa5"). InnerVolumeSpecName "kube-api-access-9prsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:27.059143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.059119 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca" pod "cc30d173-29dd-4c73-95b5-78046c770fa5" (UID: "cc30d173-29dd-4c73-95b5-78046c770fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:21:27.155418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.155337 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prsf\" (UniqueName: \"kubernetes.io/projected/cc30d173-29dd-4c73-95b5-78046c770fa5-kube-api-access-9prsf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.155418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.155376 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.155418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.155391 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc30d173-29dd-4c73-95b5-78046c770fa5-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.789573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.789397 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" event={"ID":"cc30d173-29dd-4c73-95b5-78046c770fa5","Type":"ContainerDied","Data":"48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9"} Nov 03 04:21:27.789573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.789465 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ed8a8559f2ec26c6595b5323f6d83fd107b98e5cc7d9cb0442a9bba153b6e9" Nov 03 04:21:27.790211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:27.789611 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p" Nov 03 04:21:28.037873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.036127 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 04:21:28.174541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.174503 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.174836 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.174813 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.175044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.175024 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.175242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.175224 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.175668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.175647 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.175866 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.175829 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.176043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.176025 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.176213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.176196 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.176393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.176338 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\") pod \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\" (UID: \"22dac5ae-87c1-43e7-8f76-cd6f8e462210\") " Nov 03 04:21:28.177662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.177635 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:28.177890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.177831 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:28.178049 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.178025 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:28.178205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.178188 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:28.184901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.180395 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "kube-api-access-zm5l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:28.184901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.183331 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:28.184901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.183376 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:28.184901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.183823 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:21:28.193301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.193252 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token" pod "22dac5ae-87c1-43e7-8f76-cd6f8e462210" (UID: "22dac5ae-87c1-43e7-8f76-cd6f8e462210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:28.278627 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278581 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278627 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278620 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278635 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278647 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278659 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278673 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5l9\" (UniqueName: \"kubernetes.io/projected/22dac5ae-87c1-43e7-8f76-cd6f8e462210-kube-api-access-zm5l9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278685 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278696 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.278883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.278708 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dac5ae-87c1-43e7-8f76-cd6f8e462210-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.798544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.798506 5275 generic.go:356] "Generic (PLEG): container finished" podID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerID="35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b" exitCode=0 Nov 03 04:21:28.799077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.798586 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" Nov 03 04:21:28.799077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.798595 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerDied","Data":"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b"} Nov 03 04:21:28.799077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.798687 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727" event={"ID":"22dac5ae-87c1-43e7-8f76-cd6f8e462210","Type":"ContainerDied","Data":"3e10b07d9e856ecf0ca63e2d904fd8aa66fae14db2085e4b94afcc6adb621031"} Nov 03 04:21:28.799077 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.798712 5275 scope.go:122] "RemoveContainer" containerID="35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b" Nov 03 04:21:28.810969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.810944 5275 scope.go:122] "RemoveContainer" containerID="de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26" Nov 03 04:21:28.822032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.821748 5275 scope.go:122] "RemoveContainer" containerID="35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b" Nov 03 04:21:28.825775 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:28.824314 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b\": container with ID starting with 35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b not found: ID does not exist" containerID="35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b" Nov 03 04:21:28.825775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.824359 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b"} err="failed to get container status \"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b\": rpc error: code = NotFound desc = could not find container \"35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b\": container with ID starting with 35fbd4698d27e7ed393354433c7dde284c372cdfc0a4f71fe130eb90b3f60e9b not found: ID does not exist" Nov 03 04:21:28.825775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.824384 5275 scope.go:122] "RemoveContainer" containerID="de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26" Nov 03 04:21:28.825775 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:28.824806 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26\": container with ID starting with de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26 not found: ID does not exist" containerID="de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26" Nov 03 04:21:28.825775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.824839 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26"} err="failed to get container status \"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26\": rpc error: code = NotFound desc = could not find container \"de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26\": container with ID starting with de51783cd13bf53c86cbe1dbebeea3df3be28e5c4cc74b419c1b77bdbfa8ec26 not found: ID does not exist" Nov 03 04:21:28.967047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.967003 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 04:21:28.996770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:28.996732 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cffde05-c4db-4390-b877-2bca581764da" path="/var/lib/kubelet/pods/7cffde05-c4db-4390-b877-2bca581764da/volumes" Nov 03 04:21:29.016275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:29.016236 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f3e02b-openshift-default-988d8b99f-hl727"] Nov 03 04:21:30.990434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:30.990392 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" path="/var/lib/kubelet/pods/22dac5ae-87c1-43e7-8f76-cd6f8e462210/volumes" Nov 03 04:21:31.762266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.762240 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 04:21:31.823505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.823469 5275 generic.go:356] "Generic (PLEG): container finished" podID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerID="df9a8762180db24809f0bd9f7c56e3a5ce4da524804e91662866707d076ac18e" exitCode=0 Nov 03 04:21:31.823671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.823539 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" event={"ID":"965f712d-dfe4-4139-9e6e-4012b93ab48b","Type":"ContainerDied","Data":"df9a8762180db24809f0bd9f7c56e3a5ce4da524804e91662866707d076ac18e"} Nov 03 04:21:31.825270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.825243 5275 generic.go:356] "Generic (PLEG): container finished" podID="22b20891-cb91-487d-a08a-f609f826baee" containerID="c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320" exitCode=0 Nov 03 04:21:31.825402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.825287 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" Nov 03 04:21:31.825402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.825291 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" event={"ID":"22b20891-cb91-487d-a08a-f609f826baee","Type":"ContainerDied","Data":"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320"} Nov 03 04:21:31.825402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.825343 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs" event={"ID":"22b20891-cb91-487d-a08a-f609f826baee","Type":"ContainerDied","Data":"b589b46d7000051c1d7e7cc8368a396caf9f82b99d825e09c4d7d1d41b77e231"} Nov 03 04:21:31.825402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.825375 5275 scope.go:122] "RemoveContainer" containerID="c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320" Nov 03 04:21:31.841087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.841066 5275 scope.go:122] "RemoveContainer" containerID="c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320" Nov 03 04:21:31.841423 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:31.841403 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320\": container with ID starting with c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320 not found: ID does not exist" containerID="c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320" Nov 03 04:21:31.841502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.841431 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320"} err="failed to get container status \"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320\": rpc error: code = NotFound desc = could not find container \"c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320\": container with ID starting with c8ab71c4a9a0c88bf5f7ccc4fd2133cfe5785fc1443be5244ff6522ce5ef3320 not found: ID does not exist" Nov 03 04:21:31.923623 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.923559 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 04:21:31.935839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.935810 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\" (UniqueName: \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\") pod \"22b20891-cb91-487d-a08a-f609f826baee\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " Nov 03 04:21:31.936001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.935899 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\") pod \"22b20891-cb91-487d-a08a-f609f826baee\" (UID: \"22b20891-cb91-487d-a08a-f609f826baee\") " Nov 03 04:21:31.936342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.936304 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume" pod "22b20891-cb91-487d-a08a-f609f826baee" (UID: "22b20891-cb91-487d-a08a-f609f826baee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:31.938148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:31.938123 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv" pod "22b20891-cb91-487d-a08a-f609f826baee" (UID: "22b20891-cb91-487d-a08a-f609f826baee"). InnerVolumeSpecName "kube-api-access-kd8sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:32.037193 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037152 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\") pod \"965f712d-dfe4-4139-9e6e-4012b93ab48b\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " Nov 03 04:21:32.037583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037302 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\" (UniqueName: \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\") pod \"965f712d-dfe4-4139-9e6e-4012b93ab48b\" (UID: \"965f712d-dfe4-4139-9e6e-4012b93ab48b\") " Nov 03 04:21:32.037643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037609 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume" pod "965f712d-dfe4-4139-9e6e-4012b93ab48b" (UID: "965f712d-dfe4-4139-9e6e-4012b93ab48b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:32.037749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037732 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8sv\" (UniqueName: \"kubernetes.io/projected/22b20891-cb91-487d-a08a-f609f826baee-kube-api-access-kd8sv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:32.037791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037752 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b20891-cb91-487d-a08a-f609f826baee-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:32.037791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.037762 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965f712d-dfe4-4139-9e6e-4012b93ab48b-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:32.039551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.039525 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4" pod "965f712d-dfe4-4139-9e6e-4012b93ab48b" (UID: "965f712d-dfe4-4139-9e6e-4012b93ab48b"). InnerVolumeSpecName "kube-api-access-gw6c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:32.139385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.139352 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6c4\" (UniqueName: \"kubernetes.io/projected/965f712d-dfe4-4139-9e6e-4012b93ab48b-kube-api-access-gw6c4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:32.254219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.254128 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 04:21:32.259531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.259502 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f3e02b-84d758dc6f-8h8qs"] Nov 03 04:21:32.339946 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:21:32.339908 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:21:32.836326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.836293 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" Nov 03 04:21:32.836326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.836316 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t" event={"ID":"965f712d-dfe4-4139-9e6e-4012b93ab48b","Type":"ContainerDied","Data":"a082e66310e037586d62efdc37557033444eeb01f94bf2322fb215ca82ccd2de"} Nov 03 04:21:32.836613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.836384 5275 scope.go:122] "RemoveContainer" containerID="df9a8762180db24809f0bd9f7c56e3a5ce4da524804e91662866707d076ac18e" Nov 03 04:21:32.985276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:32.985239 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b20891-cb91-487d-a08a-f609f826baee" path="/var/lib/kubelet/pods/22b20891-cb91-487d-a08a-f609f826baee/volumes" Nov 03 04:21:33.001311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:33.001264 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 04:21:33.051119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:33.051083 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f3e02b-7dd85fd499-wg46t"] Nov 03 04:21:34.049275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:34.049242 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx"] Nov 03 04:21:34.098568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:34.098528 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6335e74f-st2cx"] Nov 03 04:21:34.985084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:34.985053 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" path="/var/lib/kubelet/pods/965f712d-dfe4-4139-9e6e-4012b93ab48b/volumes" Nov 03 04:21:34.985562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:34.985547 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a536e075-1629-4612-a9ac-492c20c7e6c9" path="/var/lib/kubelet/pods/a536e075-1629-4612-a9ac-492c20c7e6c9/volumes" Nov 03 04:21:37.711145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:37.710650 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 04:21:37.711145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:37.711055 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" containerID="cri-o://6c768a9939a06584319308bfdef82f84bed63d7ab3be494c5f96fa70636b228a" gracePeriod=30 Nov 03 04:21:42.922901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:42.922863 5275 generic.go:356] "Generic (PLEG): container finished" podID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerID="6c768a9939a06584319308bfdef82f84bed63d7ab3be494c5f96fa70636b228a" exitCode=0 Nov 03 04:21:42.923522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:42.923488 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerDied","Data":"6c768a9939a06584319308bfdef82f84bed63d7ab3be494c5f96fa70636b228a"} Nov 03 04:21:42.923666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:42.923561 5275 scope.go:122] "RemoveContainer" containerID="3d8b495d1c7d9bb750bc1a40a3cfbcf1a472446c98e8a0c2276715a07b5d04a5" Nov 03 04:21:42.985563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:42.985488 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 04:21:43.060926 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.060890 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.060939 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.060959 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061133 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061181 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061261 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:43.061387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061270 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061440 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061497 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061504 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:43.061610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061552 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\") pod \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\" (UID: \"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211\") " Nov 03 04:21:43.061829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061807 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:43.061977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.061954 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:43.062078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062035 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:43.062237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062217 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.062325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062293 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.062393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062335 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.062393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062360 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.062393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.062375 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.064115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.064085 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:21:43.064311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.064273 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "kube-api-access-9tff5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:43.064436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.064309 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:21:43.064436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.064357 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token" pod "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" (UID: "bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:43.164103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.164058 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tff5\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-kube-api-access-9tff5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.164103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.164097 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.164103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.164111 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.164436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.164123 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:43.932019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.931990 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" Nov 03 04:21:43.932019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.932012 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t" event={"ID":"bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211","Type":"ContainerDied","Data":"1a2310e7fcef09a97526df90fdf5247ebec452d3eac28dd806006128f4d8aa5a"} Nov 03 04:21:43.932511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:43.932065 5275 scope.go:122] "RemoveContainer" containerID="6c768a9939a06584319308bfdef82f84bed63d7ab3be494c5f96fa70636b228a" Nov 03 04:21:44.148590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:44.148556 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 04:21:44.152763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:44.152738 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-10120d-openshift-default-577fcd6cb9-mk96t"] Nov 03 04:21:44.243875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:44.243755 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 04:21:44.354607 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:44.354527 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 04:21:44.985172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:44.985136 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" path="/var/lib/kubelet/pods/bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211/volumes" Nov 03 04:21:49.245030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:49.244966 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" containerID="cri-o://1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1" gracePeriod=25 Nov 03 04:21:49.355309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:49.355244 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" containerID="cri-o://1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308" gracePeriod=25 Nov 03 04:21:50.449660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:50.449630 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:21:50.487479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:50.487453 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:21:57.940339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:57.940283 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:21:57.940841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:21:57.940284 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:02.330958 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:02.330916 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:22:07.940954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:07.940911 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:07.940954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:07.940942 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:09.540939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.540912 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 04:22:09.580870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.580288 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\" (UniqueName: \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\") pod \"57858e7e-b33e-4d64-872b-9ccb4755cda1\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " Nov 03 04:22:09.580870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.580426 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\") pod \"57858e7e-b33e-4d64-872b-9ccb4755cda1\" (UID: \"57858e7e-b33e-4d64-872b-9ccb4755cda1\") " Nov 03 04:22:09.583890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.581562 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume" pod "57858e7e-b33e-4d64-872b-9ccb4755cda1" (UID: "57858e7e-b33e-4d64-872b-9ccb4755cda1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:09.584062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.583960 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f" pod "57858e7e-b33e-4d64-872b-9ccb4755cda1" (UID: "57858e7e-b33e-4d64-872b-9ccb4755cda1"). InnerVolumeSpecName "kube-api-access-78r4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:09.629984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.629956 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 04:22:09.682135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.682091 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\" (UniqueName: \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\") pod \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " Nov 03 04:22:09.682368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.682295 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\") pod \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\" (UID: \"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb\") " Nov 03 04:22:09.682744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.682716 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume" pod "5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" (UID: "5740fbf4-5b90-46e2-90e8-a7c295dfa4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:09.682813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.682734 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78r4f\" (UniqueName: \"kubernetes.io/projected/57858e7e-b33e-4d64-872b-9ccb4755cda1-kube-api-access-78r4f\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:09.682813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.682780 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57858e7e-b33e-4d64-872b-9ccb4755cda1-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:09.684653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.684623 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz" pod "5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" (UID: "5740fbf4-5b90-46e2-90e8-a7c295dfa4bb"). InnerVolumeSpecName "kube-api-access-m9xpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:09.783473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.783439 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9xpz\" (UniqueName: \"kubernetes.io/projected/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-kube-api-access-m9xpz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:09.783473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:09.783471 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:10.130045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.129956 5275 generic.go:356] "Generic (PLEG): container finished" podID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerID="1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308" exitCode=0 Nov 03 04:22:10.130045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.130008 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" Nov 03 04:22:10.130295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.130045 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" event={"ID":"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb","Type":"ContainerDied","Data":"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308"} Nov 03 04:22:10.130295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.130105 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm" event={"ID":"5740fbf4-5b90-46e2-90e8-a7c295dfa4bb","Type":"ContainerDied","Data":"acc43c16a491e4c1474679bda075a3b3351e6ccc2cbdd1c8e8f526ac8797f071"} Nov 03 04:22:10.130295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.130134 5275 scope.go:122] "RemoveContainer" containerID="1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308" Nov 03 04:22:10.131415 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.131392 5275 generic.go:356] "Generic (PLEG): container finished" podID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerID="1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1" exitCode=0 Nov 03 04:22:10.131521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.131425 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" Nov 03 04:22:10.131521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.131437 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" event={"ID":"57858e7e-b33e-4d64-872b-9ccb4755cda1","Type":"ContainerDied","Data":"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1"} Nov 03 04:22:10.131521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.131473 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6" event={"ID":"57858e7e-b33e-4d64-872b-9ccb4755cda1","Type":"ContainerDied","Data":"da7b0288d0fec2c069ae5e7a4cd9e0212c73ea1c08bc714bd982062363a56377"} Nov 03 04:22:10.145364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.145341 5275 scope.go:122] "RemoveContainer" containerID="1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308" Nov 03 04:22:10.145697 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:10.145675 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308\": container with ID starting with 1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308 not found: ID does not exist" containerID="1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308" Nov 03 04:22:10.145782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.145705 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308"} err="failed to get container status \"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308\": rpc error: code = NotFound desc = could not find container \"1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308\": container with ID starting with 1704c6de57d1f74d1ef020e178c0f3673a8513de350b846566ae06bade28b308 not found: ID does not exist" Nov 03 04:22:10.145782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.145722 5275 scope.go:122] "RemoveContainer" containerID="1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1" Nov 03 04:22:10.155157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.155139 5275 scope.go:122] "RemoveContainer" containerID="1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1" Nov 03 04:22:10.155477 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:10.155456 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1\": container with ID starting with 1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1 not found: ID does not exist" containerID="1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1" Nov 03 04:22:10.155564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.155486 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1"} err="failed to get container status \"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1\": rpc error: code = NotFound desc = could not find container \"1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1\": container with ID starting with 1b2223fe5d3fbd5a8b963206159f4bf87331e4617c7cd30b4a7d44ea5f6fbab1 not found: ID does not exist" Nov 03 04:22:10.296511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.296476 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 04:22:10.344370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.344332 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9daccc-76464bfb55-7v2hm"] Nov 03 04:22:10.407802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.407711 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 04:22:10.412140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.412109 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9daccc-5f6d464f9-pzjj6"] Nov 03 04:22:10.986162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.986128 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" path="/var/lib/kubelet/pods/5740fbf4-5b90-46e2-90e8-a7c295dfa4bb/volumes" Nov 03 04:22:10.986595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:10.986533 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" path="/var/lib/kubelet/pods/57858e7e-b33e-4d64-872b-9ccb4755cda1/volumes" Nov 03 04:22:29.266072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266041 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d"] Nov 03 04:22:29.266560 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266545 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:29.266610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266562 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:29.266610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266574 5275 cpu_manager.go:414] "containerMap: removing container" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" Nov 03 04:22:29.266610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266579 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" Nov 03 04:22:29.266610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266588 5275 cpu_manager.go:414] "containerMap: removing container" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" Nov 03 04:22:29.266610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266594 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266614 5275 cpu_manager.go:414] "containerMap: removing container" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266619 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266625 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266630 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266636 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266640 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266646 5275 cpu_manager.go:414] "containerMap: removing container" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266652 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266665 5275 cpu_manager.go:414] "containerMap: removing container" podUID="cc30d173-29dd-4c73-95b5-78046c770fa5" containerName="revoke-keys" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266669 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc30d173-29dd-4c73-95b5-78046c770fa5" containerName="revoke-keys" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266681 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266686 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266697 5275 cpu_manager.go:414] "containerMap: removing container" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266704 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266716 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266720 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266792 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:29.266795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266800 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b20891-cb91-487d-a08a-f609f826baee" containerName="payload-processing" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266808 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="965f712d-dfe4-4139-9e6e-4012b93ab48b" containerName="payload-pre-processing" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266816 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266821 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5740fbf4-5b90-46e2-90e8-a7c295dfa4bb" containerName="payload-pre-processing" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266827 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266831 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dac5ae-87c1-43e7-8f76-cd6f8e462210" containerName="istio-proxy" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266859 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc30d173-29dd-4c73-95b5-78046c770fa5" containerName="revoke-keys" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266869 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="57858e7e-b33e-4d64-872b-9ccb4755cda1" containerName="payload-processing" Nov 03 04:22:29.267371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.266877 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cffde05-c4db-4390-b877-2bca581764da" containerName="istio-proxy" Nov 03 04:22:29.270143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.270125 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.323581 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.323503 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:22:29.373421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.373377 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:22:29.373624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.373428 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:22:29.373624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.373502 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:22:29.377954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.377926 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d"] Nov 03 04:22:29.411957 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.411918 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.412166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.411996 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h8g\" (UniqueName: \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.412166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.412085 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.513068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.513036 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h8g\" (UniqueName: \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.513261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.513086 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.513261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.513240 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.513754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.513727 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.515911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.515891 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.573923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.573874 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h8g\" (UniqueName: \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\") pod \"maas-api-revoke-sub-47fe86ea-2xp7d\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:29.583711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:29.583683 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:30.004983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:30.004956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d"] Nov 03 04:22:30.007449 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:22:30.007422 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1052bd4_cd1c_47f4_aea2_11969daf75f6.slice/crio-9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0 WatchSource:0}: Error finding container 9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0: Status 404 returned error can't find the container with id 9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0 Nov 03 04:22:30.274532 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:30.274492 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" event={"ID":"a1052bd4-cd1c-47f4-aea2-11969daf75f6","Type":"ContainerStarted","Data":"9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0"} Nov 03 04:22:31.140512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.140475 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv"] Nov 03 04:22:31.141160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.141139 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbbdb26-48e9-49dd-ad96-4d9ef1ee2211" containerName="istio-proxy" Nov 03 04:22:31.144370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.144353 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.202384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.202339 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv"] Nov 03 04:22:31.236470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.236432 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.236683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.236481 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6drw\" (UniqueName: \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.236683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.236520 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.283382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.283351 5275 generic.go:356] "Generic (PLEG): container finished" podID="a1052bd4-cd1c-47f4-aea2-11969daf75f6" containerID="39aa4e3d63e49490791f7ee3d28909aa6da2db89b006efdabb4549a5b93b2e7d" exitCode=0 Nov 03 04:22:31.283811 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.283445 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" event={"ID":"a1052bd4-cd1c-47f4-aea2-11969daf75f6","Type":"ContainerDied","Data":"39aa4e3d63e49490791f7ee3d28909aa6da2db89b006efdabb4549a5b93b2e7d"} Nov 03 04:22:31.337788 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.337754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.337788 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.337802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6drw\" (UniqueName: \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.338060 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.337864 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.338601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.338578 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.340366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.340337 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.398971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.398936 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6drw\" (UniqueName: \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\") pod \"maas-api-revoke-sub-c8b49f89-6fshv\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.459917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.459839 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:31.840861 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:31.840818 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv"] Nov 03 04:22:31.841989 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:22:31.841964 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32052f15_57b6_41f5_981c_297a45b04a21.slice/crio-054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340 WatchSource:0}: Error finding container 054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340: Status 404 returned error can't find the container with id 054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340 Nov 03 04:22:32.290812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.290777 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" event={"ID":"32052f15-57b6-41f5-981c-297a45b04a21","Type":"ContainerStarted","Data":"f89efe19b7c90c0dca2381379486f28ef330b006c047f3a26343e19deaf98e54"} Nov 03 04:22:32.291288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.290833 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" event={"ID":"32052f15-57b6-41f5-981c-297a45b04a21","Type":"ContainerStarted","Data":"054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340"} Nov 03 04:22:32.333790 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:32.333749 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:22:32.410964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.410814 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" podStartSLOduration=1.410793993 podStartE2EDuration="1.410793993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:22:32.406914295 +0000 UTC m=+4843.559602721" watchObservedRunningTime="2026-11-03 04:22:32.410793993 +0000 UTC m=+4843.563482415" Nov 03 04:22:32.464824 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.464797 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:32.558442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.558405 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\" (UniqueName: \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\") pod \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " Nov 03 04:22:32.558625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.558540 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\") pod \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " Nov 03 04:22:32.558625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.558568 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\") pod \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\" (UID: \"a1052bd4-cd1c-47f4-aea2-11969daf75f6\") " Nov 03 04:22:32.558883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.558839 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca" pod "a1052bd4-cd1c-47f4-aea2-11969daf75f6" (UID: "a1052bd4-cd1c-47f4-aea2-11969daf75f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:32.559022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.559005 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:32.560950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.560920 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca" pod "a1052bd4-cd1c-47f4-aea2-11969daf75f6" (UID: "a1052bd4-cd1c-47f4-aea2-11969daf75f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:32.560950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.560932 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g" pod "a1052bd4-cd1c-47f4-aea2-11969daf75f6" (UID: "a1052bd4-cd1c-47f4-aea2-11969daf75f6"). InnerVolumeSpecName "kube-api-access-d5h8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:32.659840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.659804 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5h8g\" (UniqueName: \"kubernetes.io/projected/a1052bd4-cd1c-47f4-aea2-11969daf75f6-kube-api-access-d5h8g\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:32.659840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:32.659839 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1052bd4-cd1c-47f4-aea2-11969daf75f6-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:33.298291 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:33.298237 5275 generic.go:356] "Generic (PLEG): container finished" podID="32052f15-57b6-41f5-981c-297a45b04a21" containerID="f89efe19b7c90c0dca2381379486f28ef330b006c047f3a26343e19deaf98e54" exitCode=0 Nov 03 04:22:33.298717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:33.298309 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" event={"ID":"32052f15-57b6-41f5-981c-297a45b04a21","Type":"ContainerDied","Data":"f89efe19b7c90c0dca2381379486f28ef330b006c047f3a26343e19deaf98e54"} Nov 03 04:22:33.300025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:33.299997 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" event={"ID":"a1052bd4-cd1c-47f4-aea2-11969daf75f6","Type":"ContainerDied","Data":"9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0"} Nov 03 04:22:33.300156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:33.300028 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d" Nov 03 04:22:33.300156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:33.300048 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cf87f31d824ad65f5922eff4cb8fa642eefa8dba7dbb68a91d3fd77b149d3e0" Nov 03 04:22:34.463706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.463677 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:34.583169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.583082 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\") pod \"32052f15-57b6-41f5-981c-297a45b04a21\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " Nov 03 04:22:34.583326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.583211 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\" (UniqueName: \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\") pod \"32052f15-57b6-41f5-981c-297a45b04a21\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " Nov 03 04:22:34.583326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.583312 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\") pod \"32052f15-57b6-41f5-981c-297a45b04a21\" (UID: \"32052f15-57b6-41f5-981c-297a45b04a21\") " Nov 03 04:22:34.583471 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.583447 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca" pod "32052f15-57b6-41f5-981c-297a45b04a21" (UID: "32052f15-57b6-41f5-981c-297a45b04a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:34.583655 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.583635 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:34.585534 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.585504 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca" pod "32052f15-57b6-41f5-981c-297a45b04a21" (UID: "32052f15-57b6-41f5-981c-297a45b04a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:34.585662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.585585 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw" pod "32052f15-57b6-41f5-981c-297a45b04a21" (UID: "32052f15-57b6-41f5-981c-297a45b04a21"). InnerVolumeSpecName "kube-api-access-n6drw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:34.684989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.684954 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6drw\" (UniqueName: \"kubernetes.io/projected/32052f15-57b6-41f5-981c-297a45b04a21-kube-api-access-n6drw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:34.684989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.684984 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32052f15-57b6-41f5-981c-297a45b04a21-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:34.948755 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.948671 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 04:22:34.956246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.956216 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 04:22:34.997810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:34.997780 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 04:22:35.314512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:35.314472 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" event={"ID":"32052f15-57b6-41f5-981c-297a45b04a21","Type":"ContainerDied","Data":"054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340"} Nov 03 04:22:35.314512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:35.314505 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv" Nov 03 04:22:35.314750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:35.314561 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054b73d926743551cb62aac87cececded42fd69ce7f32b53b1706c47257f9340" Nov 03 04:22:41.049152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.049112 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7"] Nov 03 04:22:41.098794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.098757 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7093c7-tj4j7"] Nov 03 04:22:41.295262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.295204 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4"] Nov 03 04:22:41.296469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.296440 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a1052bd4-cd1c-47f4-aea2-11969daf75f6" containerName="revoke-keys" Nov 03 04:22:41.296654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.296637 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1052bd4-cd1c-47f4-aea2-11969daf75f6" containerName="revoke-keys" Nov 03 04:22:41.296792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.296781 5275 cpu_manager.go:414] "containerMap: removing container" podUID="32052f15-57b6-41f5-981c-297a45b04a21" containerName="revoke-keys" Nov 03 04:22:41.297131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.297109 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32052f15-57b6-41f5-981c-297a45b04a21" containerName="revoke-keys" Nov 03 04:22:41.297832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.297809 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1052bd4-cd1c-47f4-aea2-11969daf75f6" containerName="revoke-keys" Nov 03 04:22:41.297832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.297837 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="32052f15-57b6-41f5-981c-297a45b04a21" containerName="revoke-keys" Nov 03 04:22:41.305796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.305718 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.356636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.356599 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4"] Nov 03 04:22:41.359180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.359153 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:22:41.359180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.359175 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:22:41.359419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.359247 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:22:41.402094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.402064 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:22:41.463340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.463297 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.463564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.463481 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqs5t\" (UniqueName: \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.463643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.463593 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.564380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.564276 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.564380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.564380 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqs5t\" (UniqueName: \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.564646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.564426 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.565004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.564973 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.567157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.567120 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.625087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.625059 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqs5t\" (UniqueName: \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\") pod \"maas-api-revoke-sub-d56a73db-l67h4\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:41.922108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:41.922021 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:42.305483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:42.305457 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4"] Nov 03 04:22:42.307704 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:22:42.307670 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949a6d65_f8a6_4cd7_b495_369703bbf310.slice/crio-1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67 WatchSource:0}: Error finding container 1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67: Status 404 returned error can't find the container with id 1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67 Nov 03 04:22:42.362436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:42.362403 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" event={"ID":"949a6d65-f8a6-4cd7-b495-369703bbf310","Type":"ContainerStarted","Data":"1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67"} Nov 03 04:22:42.839039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:42.839002 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 04:22:42.943519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:42.943427 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 04:22:42.984642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:42.984611 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdaf8892-2dc4-45f5-8212-5dd44ebbb48e" path="/var/lib/kubelet/pods/fdaf8892-2dc4-45f5-8212-5dd44ebbb48e/volumes" Nov 03 04:22:43.372146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:43.372113 5275 generic.go:356] "Generic (PLEG): container finished" podID="949a6d65-f8a6-4cd7-b495-369703bbf310" containerID="5e23cc848ce84f2915e14b108db90f04f934a923f28ec6ac9d9b0239f0506a1c" exitCode=0 Nov 03 04:22:43.372587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:43.372212 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" event={"ID":"949a6d65-f8a6-4cd7-b495-369703bbf310","Type":"ContainerDied","Data":"5e23cc848ce84f2915e14b108db90f04f934a923f28ec6ac9d9b0239f0506a1c"} Nov 03 04:22:44.576255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.576229 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:44.603239 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.603204 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\" (UniqueName: \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\") pod \"949a6d65-f8a6-4cd7-b495-369703bbf310\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " Nov 03 04:22:44.603418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.603279 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\") pod \"949a6d65-f8a6-4cd7-b495-369703bbf310\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " Nov 03 04:22:44.603418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.603389 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\") pod \"949a6d65-f8a6-4cd7-b495-369703bbf310\" (UID: \"949a6d65-f8a6-4cd7-b495-369703bbf310\") " Nov 03 04:22:44.603778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.603749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca" pod "949a6d65-f8a6-4cd7-b495-369703bbf310" (UID: "949a6d65-f8a6-4cd7-b495-369703bbf310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:44.605880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.605829 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t" pod "949a6d65-f8a6-4cd7-b495-369703bbf310" (UID: "949a6d65-f8a6-4cd7-b495-369703bbf310"). InnerVolumeSpecName "kube-api-access-jqs5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:44.605880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.605867 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca" pod "949a6d65-f8a6-4cd7-b495-369703bbf310" (UID: "949a6d65-f8a6-4cd7-b495-369703bbf310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:44.680802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.680711 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb"] Nov 03 04:22:44.681293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.681278 5275 cpu_manager.go:414] "containerMap: removing container" podUID="949a6d65-f8a6-4cd7-b495-369703bbf310" containerName="revoke-keys" Nov 03 04:22:44.681341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.681295 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949a6d65-f8a6-4cd7-b495-369703bbf310" containerName="revoke-keys" Nov 03 04:22:44.681423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.681411 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="949a6d65-f8a6-4cd7-b495-369703bbf310" containerName="revoke-keys" Nov 03 04:22:44.684797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.684778 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.705198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705163 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.705402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705382 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.705489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705442 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hfg\" (UniqueName: \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.705591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705570 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:44.705650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705595 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949a6d65-f8a6-4cd7-b495-369703bbf310-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:44.705650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.705606 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqs5t\" (UniqueName: \"kubernetes.io/projected/949a6d65-f8a6-4cd7-b495-369703bbf310-kube-api-access-jqs5t\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:44.758601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.758564 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb"] Nov 03 04:22:44.806692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.806657 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.806918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.806714 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hfg\" (UniqueName: \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.806918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.806761 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.807372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.807343 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.809337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.809315 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:44.873449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:44.873415 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hfg\" (UniqueName: \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\") pod \"maas-api-revoke-sub-85e3fca9-clcmb\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:45.000528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.000503 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:45.213314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.213279 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 04:22:45.322980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.322931 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 04:22:45.382757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.382728 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb"] Nov 03 04:22:45.383536 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:22:45.383505 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cb0df3_e754_418e_8091_47e99fd4a5a3.slice/crio-348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5 WatchSource:0}: Error finding container 348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5: Status 404 returned error can't find the container with id 348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5 Nov 03 04:22:45.393365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.393339 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" Nov 03 04:22:45.393507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.393369 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4" event={"ID":"949a6d65-f8a6-4cd7-b495-369703bbf310","Type":"ContainerDied","Data":"1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67"} Nov 03 04:22:45.393507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:45.393420 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a03b648f6b2201c417df40d72001679e884bbad41369757486aeb514f14cd67" Nov 03 04:22:46.401749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:46.401705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" event={"ID":"07cb0df3-e754-418e-8091-47e99fd4a5a3","Type":"ContainerStarted","Data":"df04397437782fea693bae1f0ff984d0d7d89e41def936fd98166ab4f77807d7"} Nov 03 04:22:46.402178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:46.401763 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" event={"ID":"07cb0df3-e754-418e-8091-47e99fd4a5a3","Type":"ContainerStarted","Data":"348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5"} Nov 03 04:22:46.518152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:46.518090 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" podStartSLOduration=2.518074058 podStartE2EDuration="2.518074058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:22:46.516677251 +0000 UTC m=+4857.669365672" watchObservedRunningTime="2026-11-03 04:22:46.518074058 +0000 UTC m=+4857.670762478" Nov 03 04:22:47.410880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:47.410821 5275 generic.go:356] "Generic (PLEG): container finished" podID="07cb0df3-e754-418e-8091-47e99fd4a5a3" containerID="df04397437782fea693bae1f0ff984d0d7d89e41def936fd98166ab4f77807d7" exitCode=0 Nov 03 04:22:47.411345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:47.410909 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" event={"ID":"07cb0df3-e754-418e-8091-47e99fd4a5a3","Type":"ContainerDied","Data":"df04397437782fea693bae1f0ff984d0d7d89e41def936fd98166ab4f77807d7"} Nov 03 04:22:47.840405 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:47.840335 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" containerID="cri-o://e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65" gracePeriod=25 Nov 03 04:22:47.944212 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:47.944166 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" containerID="cri-o://7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961" gracePeriod=25 Nov 03 04:22:48.578703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.578672 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:48.650855 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.650808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\") pod \"07cb0df3-e754-418e-8091-47e99fd4a5a3\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " Nov 03 04:22:48.651051 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.650876 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\" (UniqueName: \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\") pod \"07cb0df3-e754-418e-8091-47e99fd4a5a3\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " Nov 03 04:22:48.651051 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.650980 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\") pod \"07cb0df3-e754-418e-8091-47e99fd4a5a3\" (UID: \"07cb0df3-e754-418e-8091-47e99fd4a5a3\") " Nov 03 04:22:48.651249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.651227 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca" pod "07cb0df3-e754-418e-8091-47e99fd4a5a3" (UID: "07cb0df3-e754-418e-8091-47e99fd4a5a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:48.651865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.651822 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.653247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.653217 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca" pod "07cb0df3-e754-418e-8091-47e99fd4a5a3" (UID: "07cb0df3-e754-418e-8091-47e99fd4a5a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:48.653372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.653299 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg" pod "07cb0df3-e754-418e-8091-47e99fd4a5a3" (UID: "07cb0df3-e754-418e-8091-47e99fd4a5a3"). InnerVolumeSpecName "kube-api-access-m2hfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:48.753409 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.753304 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hfg\" (UniqueName: \"kubernetes.io/projected/07cb0df3-e754-418e-8091-47e99fd4a5a3-kube-api-access-m2hfg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.753409 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:48.753346 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cb0df3-e754-418e-8091-47e99fd4a5a3-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:49.427522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.427488 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" Nov 03 04:22:49.427705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.427521 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb" event={"ID":"07cb0df3-e754-418e-8091-47e99fd4a5a3","Type":"ContainerDied","Data":"348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5"} Nov 03 04:22:49.427705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.427577 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="348700b3a5ac3c2f2c8f6c0bcb5784239e4ddfe9c2790b37c63e599ca109a1b5" Nov 03 04:22:49.963653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.963613 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:49.964101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.963737 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:49.970358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.970304 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" containerID="cri-o://b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab" gracePeriod=45 Nov 03 04:22:49.979356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:49.979320 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" containerID="cri-o://df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0" gracePeriod=45 Nov 03 04:22:50.022109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.022070 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" containerID="cri-o://99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a" gracePeriod=45 Nov 03 04:22:50.218950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.218786 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" containerID="cri-o://2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5" gracePeriod=25 Nov 03 04:22:50.311152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.311124 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 04:22:50.324262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.324228 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" containerID="cri-o://d1909d9f29a0550e6993767d61a1697dac9e5906e6246486cf4684db72d36571" gracePeriod=25 Nov 03 04:22:50.340238 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.340211 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 04:22:50.342336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.342312 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 04:22:50.374657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374622 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.374657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374668 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.374958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374724 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.374958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374761 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.374958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374794 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\" (UniqueName: \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.374958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374869 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.374958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374919 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374960 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.374989 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\" (UniqueName: \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375122 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\" (UniqueName: \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375160 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375202 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.375245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375243 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375256 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375274 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375306 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\" (UniqueName: \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\") pod \"456f1b37-ed97-4514-932c-141a33ade80e\" (UID: \"456f1b37-ed97-4514-932c-141a33ade80e\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375350 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\") pod \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\" (UID: \"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375376 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375407 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\" (UniqueName: \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\") pod \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\" (UID: \"35e43148-cf6e-49cc-9c1c-1920b54edd8c\") " Nov 03 04:22:50.375552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375482 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.376002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.375977 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.376432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.376227 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.376432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.376273 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.376628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.376483 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.377131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.376894 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.377240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.377188 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.377240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.377222 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.377437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.377401 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.377949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.377917 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.378180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.378159 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.379032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.378996 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "kube-api-access-59zjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:50.379132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.379108 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "kube-api-access-j8pl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:50.380622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.380596 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "kube-api-access-szwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:50.380930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.380890 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:50.381072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.381049 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.381792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.381756 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm" pod "35e43148-cf6e-49cc-9c1c-1920b54edd8c" (UID: "35e43148-cf6e-49cc-9c1c-1920b54edd8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:50.382017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.381992 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs" pod "456f1b37-ed97-4514-932c-141a33ade80e" (UID: "456f1b37-ed97-4514-932c-141a33ade80e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:50.382462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.382426 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs" pod "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" (UID: "20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:50.439027 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.438977 5275 generic.go:356] "Generic (PLEG): container finished" podID="456f1b37-ed97-4514-932c-141a33ade80e" containerID="df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0" exitCode=0 Nov 03 04:22:50.439205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.439071 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" event={"ID":"456f1b37-ed97-4514-932c-141a33ade80e","Type":"ContainerDied","Data":"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0"} Nov 03 04:22:50.439205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.439109 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" event={"ID":"456f1b37-ed97-4514-932c-141a33ade80e","Type":"ContainerDied","Data":"d649b3f5d7e5c8475da803285c0704d798b80e75044b13b746a5782e4c8de028"} Nov 03 04:22:50.439205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.439132 5275 scope.go:122] "RemoveContainer" containerID="df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0" Nov 03 04:22:50.439391 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.439295 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4" Nov 03 04:22:50.446606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.446573 5275 generic.go:356] "Generic (PLEG): container finished" podID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerID="b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab" exitCode=0 Nov 03 04:22:50.446761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.446689 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" event={"ID":"35e43148-cf6e-49cc-9c1c-1920b54edd8c","Type":"ContainerDied","Data":"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab"} Nov 03 04:22:50.446761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.446726 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" event={"ID":"35e43148-cf6e-49cc-9c1c-1920b54edd8c","Type":"ContainerDied","Data":"67fcf91ab9453d0a48a39fcb4f747bc0ca464aaf25d3368b2a4dc831d72939b7"} Nov 03 04:22:50.446936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.446806 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh" Nov 03 04:22:50.449555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.449522 5275 generic.go:356] "Generic (PLEG): container finished" podID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerID="99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a" exitCode=0 Nov 03 04:22:50.449700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.449677 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" event={"ID":"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af","Type":"ContainerDied","Data":"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a"} Nov 03 04:22:50.449775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.449714 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" event={"ID":"20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af","Type":"ContainerDied","Data":"960214dd651f2edba2c8171506f6c92a1a2868b8fe5fd84697cf8956d24f4373"} Nov 03 04:22:50.449840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.449774 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc" Nov 03 04:22:50.456210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.456124 5275 scope.go:122] "RemoveContainer" containerID="df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0" Nov 03 04:22:50.456460 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:50.456432 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0\": container with ID starting with df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0 not found: ID does not exist" containerID="df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0" Nov 03 04:22:50.456593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.456470 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0"} err="failed to get container status \"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0\": rpc error: code = NotFound desc = could not find container \"df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0\": container with ID starting with df942e08ca567f8074c7bb4c8ef7c4b758f6cc357209b12c6677b130f4e72ac0 not found: ID does not exist" Nov 03 04:22:50.456593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.456488 5275 scope.go:122] "RemoveContainer" containerID="b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab" Nov 03 04:22:50.469998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.469971 5275 scope.go:122] "RemoveContainer" containerID="b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab" Nov 03 04:22:50.470359 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:50.470335 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab\": container with ID starting with b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab not found: ID does not exist" containerID="b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab" Nov 03 04:22:50.470472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.470367 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab"} err="failed to get container status \"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab\": rpc error: code = NotFound desc = could not find container \"b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab\": container with ID starting with b9d9068444fd621fce4fe439accb404a15e3492905e6990ede2a5b63fad4b2ab not found: ID does not exist" Nov 03 04:22:50.470472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.470384 5275 scope.go:122] "RemoveContainer" containerID="99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478133 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478162 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478177 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478192 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478206 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478219 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478234 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwnl\" (UniqueName: \"kubernetes.io/projected/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-kube-api-access-szwnl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478248 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pl6\" (UniqueName: \"kubernetes.io/projected/456f1b37-ed97-4514-932c-141a33ade80e-kube-api-access-j8pl6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478262 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478276 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456f1b37-ed97-4514-932c-141a33ade80e-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478289 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478302 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478314 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456f1b37-ed97-4514-932c-141a33ade80e-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478326 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478338 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35e43148-cf6e-49cc-9c1c-1920b54edd8c-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.478375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.478350 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59zjp\" (UniqueName: \"kubernetes.io/projected/35e43148-cf6e-49cc-9c1c-1920b54edd8c-kube-api-access-59zjp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:50.481710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.481683 5275 scope.go:122] "RemoveContainer" containerID="99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a" Nov 03 04:22:50.482111 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:22:50.482082 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a\": container with ID starting with 99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a not found: ID does not exist" containerID="99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a" Nov 03 04:22:50.482194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.482123 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a"} err="failed to get container status \"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a\": rpc error: code = NotFound desc = could not find container \"99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a\": container with ID starting with 99959a08401b599d5d85a14a7716bed300d293864f30f742da351892c209ae7a not found: ID does not exist" Nov 03 04:22:50.772979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.772909 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 04:22:50.775954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.775926 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/simulator-w4-b1ad4e-kserve-d66677d94-wj5xc"] Nov 03 04:22:50.883984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.883945 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 04:22:50.889214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.889183 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-w4-b1ad4e-kserve-7f45b84bcd-4ktjh"] Nov 03 04:22:50.985384 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.985346 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" path="/var/lib/kubelet/pods/20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af/volumes" Nov 03 04:22:50.985965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.985947 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" path="/var/lib/kubelet/pods/35e43148-cf6e-49cc-9c1c-1920b54edd8c/volumes" Nov 03 04:22:50.999450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:50.999417 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 04:22:51.002056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:51.002034 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/premium-w4-b1ad4e-kserve-d8d94b579-mvzn4"] Nov 03 04:22:52.984170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:52.984081 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456f1b37-ed97-4514-932c-141a33ade80e" path="/var/lib/kubelet/pods/456f1b37-ed97-4514-932c-141a33ade80e/volumes" Nov 03 04:22:54.437300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:54.437266 5275 scope.go:122] "RemoveContainer" containerID="ebb9c84a14087cc6cca1c9ea9ecd8357b50f1a585b4d6517f98ac503c63b8a12" Nov 03 04:22:56.943795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:56.943753 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:57.948504 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:57.948468 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:58.049887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:58.049830 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5"] Nov 03 04:22:58.056382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:58.056349 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca41d644-lxgt5"] Nov 03 04:22:58.984870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:58.984815 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686c1465-24c0-4f0d-95d6-22ef678caa63" path="/var/lib/kubelet/pods/686c1465-24c0-4f0d-95d6-22ef678caa63/volumes" Nov 03 04:22:59.963054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:59.963016 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:22:59.965181 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:22:59.963060 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:23:02.330255 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:02.330218 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:23:05.006931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.006896 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:05.007435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007412 5275 cpu_manager.go:414] "containerMap: removing container" podUID="07cb0df3-e754-418e-8091-47e99fd4a5a3" containerName="revoke-keys" Nov 03 04:23:05.007435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007427 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07cb0df3-e754-418e-8091-47e99fd4a5a3" containerName="revoke-keys" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007446 5275 cpu_manager.go:414] "containerMap: removing container" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007451 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007459 5275 cpu_manager.go:414] "containerMap: removing container" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007466 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007472 5275 cpu_manager.go:414] "containerMap: removing container" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007478 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007545 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="456f1b37-ed97-4514-932c-141a33ade80e" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007555 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e43148-cf6e-49cc-9c1c-1920b54edd8c" containerName="main" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007568 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="07cb0df3-e754-418e-8091-47e99fd4a5a3" containerName="revoke-keys" Nov 03 04:23:05.007609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.007576 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f01655-4ce7-4fbe-aa5d-8cbcbfc9a3af" containerName="main" Nov 03 04:23:05.011073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.011056 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.063229 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.063192 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\"" Nov 03 04:23:05.116412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.116375 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:05.137833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.137795 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.137838 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.137957 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ndw\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138070 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138106 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138142 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138246 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138391 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.138551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.138500 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.239799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.239754 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.239819 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.239910 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240001 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240064 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240113 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240149 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240176 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ndw\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240239 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240278 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.240588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240493 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.241003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.240955 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.241141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.241018 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.241288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.241262 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.243032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.243008 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.243146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.243085 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.301751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.301681 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.301751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.301724 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ndw\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.324997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.324968 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:05.767293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.767262 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:05.769338 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:23:05.769310 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d071ba_c366_4047_bbe1_c4d4af7bc7c9.slice/crio-5dd05b2f371d7814cddeedb0a9f07d7d7bca281de7f2123b18e740961ec18018 WatchSource:0}: Error finding container 5dd05b2f371d7814cddeedb0a9f07d7d7bca281de7f2123b18e740961ec18018: Status 404 returned error can't find the container with id 5dd05b2f371d7814cddeedb0a9f07d7d7bca281de7f2123b18e740961ec18018 Nov 03 04:23:05.772529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.772487 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:05.772618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.772580 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:05.772662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:05.772623 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:06.596601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:06.596559 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" event={"ID":"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9","Type":"ContainerStarted","Data":"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403"} Nov 03 04:23:06.597112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:06.596621 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" event={"ID":"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9","Type":"ContainerStarted","Data":"5dd05b2f371d7814cddeedb0a9f07d7d7bca281de7f2123b18e740961ec18018"} Nov 03 04:23:06.746519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:06.746456 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podStartSLOduration=2.746437916 podStartE2EDuration="2.746437916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:23:06.742523135 +0000 UTC m=+4877.895211557" watchObservedRunningTime="2026-11-03 04:23:06.746437916 +0000 UTC m=+4877.899126342" Nov 03 04:23:06.943760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:06.943648 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:23:07.326507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:07.326472 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:07.328200 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:07.328173 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:07.328333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:07.328263 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:07.950657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:07.948960 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:23:08.188465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.188430 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 04:23:08.296231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.296204 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 04:23:08.301186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301164 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\") pod \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " Nov 03 04:23:08.301313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301254 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\") pod \"952569b9-4a82-4650-82e7-79de667bbd6f\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " Nov 03 04:23:08.301313 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301288 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\") pod \"952569b9-4a82-4650-82e7-79de667bbd6f\" (UID: \"952569b9-4a82-4650-82e7-79de667bbd6f\") " Nov 03 04:23:08.301434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301318 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\" (UniqueName: \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\") pod \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\" (UID: \"5ef707fe-4249-4ad5-a27f-265370cb9ef3\") " Nov 03 04:23:08.301586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301562 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume" pod "5ef707fe-4249-4ad5-a27f-265370cb9ef3" (UID: "5ef707fe-4249-4ad5-a27f-265370cb9ef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:08.301694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301669 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume" pod "952569b9-4a82-4650-82e7-79de667bbd6f" (UID: "952569b9-4a82-4650-82e7-79de667bbd6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:08.301766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.301695 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef707fe-4249-4ad5-a27f-265370cb9ef3-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:08.303565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.303537 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc" pod "952569b9-4a82-4650-82e7-79de667bbd6f" (UID: "952569b9-4a82-4650-82e7-79de667bbd6f"). InnerVolumeSpecName "kube-api-access-8c7tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:08.304173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.304144 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n" pod "5ef707fe-4249-4ad5-a27f-265370cb9ef3" (UID: "5ef707fe-4249-4ad5-a27f-265370cb9ef3"). InnerVolumeSpecName "kube-api-access-w672n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:08.326563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.326200 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:08.326563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.326257 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:08.402388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.402303 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/952569b9-4a82-4650-82e7-79de667bbd6f-kube-api-access-8c7tc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:08.402388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.402337 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952569b9-4a82-4650-82e7-79de667bbd6f-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:08.402388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.402347 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w672n\" (UniqueName: \"kubernetes.io/projected/5ef707fe-4249-4ad5-a27f-265370cb9ef3-kube-api-access-w672n\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:08.610201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.610167 5275 generic.go:356] "Generic (PLEG): container finished" podID="952569b9-4a82-4650-82e7-79de667bbd6f" containerID="7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961" exitCode=0 Nov 03 04:23:08.610424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.610235 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" Nov 03 04:23:08.610424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.610256 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" event={"ID":"952569b9-4a82-4650-82e7-79de667bbd6f","Type":"ContainerDied","Data":"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961"} Nov 03 04:23:08.610424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.610316 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v" event={"ID":"952569b9-4a82-4650-82e7-79de667bbd6f","Type":"ContainerDied","Data":"aa54e8fdba86e5a993e76bc33a7e92d2fc118d16a7f444424ff70bab6389a900"} Nov 03 04:23:08.610424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.610345 5275 scope.go:122] "RemoveContainer" containerID="7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961" Nov 03 04:23:08.611990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.611924 5275 generic.go:356] "Generic (PLEG): container finished" podID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerID="e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65" exitCode=0 Nov 03 04:23:08.612503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.612486 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" Nov 03 04:23:08.619889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.619862 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" event={"ID":"5ef707fe-4249-4ad5-a27f-265370cb9ef3","Type":"ContainerDied","Data":"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65"} Nov 03 04:23:08.620017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.619919 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426" event={"ID":"5ef707fe-4249-4ad5-a27f-265370cb9ef3","Type":"ContainerDied","Data":"f48b4a4f0d7bbb17990b979410f9a98f9a348bd5be0d4598aef908ea9601a00b"} Nov 03 04:23:08.621213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.621190 5275 scope.go:122] "RemoveContainer" containerID="7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961" Nov 03 04:23:08.621578 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:08.621551 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961\": container with ID starting with 7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961 not found: ID does not exist" containerID="7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961" Nov 03 04:23:08.621655 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.621588 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961"} err="failed to get container status \"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961\": rpc error: code = NotFound desc = could not find container \"7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961\": container with ID starting with 7a1823581f5e60a9bc4c1496a82010612c37b2e32bd03bdc71c02e71f3fe2961 not found: ID does not exist" Nov 03 04:23:08.621655 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.621606 5275 scope.go:122] "RemoveContainer" containerID="e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65" Nov 03 04:23:08.631236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.631208 5275 scope.go:122] "RemoveContainer" containerID="e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65" Nov 03 04:23:08.631538 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:08.631520 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65\": container with ID starting with e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65 not found: ID does not exist" containerID="e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65" Nov 03 04:23:08.631599 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.631547 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65"} err="failed to get container status \"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65\": rpc error: code = NotFound desc = could not find container \"e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65\": container with ID starting with e6fe1e094e934a633aba13d44e27c7a2b84380e43de4e22196a8cc5ce6026d65 not found: ID does not exist" Nov 03 04:23:08.782124 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.782088 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 04:23:08.785695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.785670 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b1ad4e-fd99d4fc-9g426"] Nov 03 04:23:08.892971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.892935 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 04:23:08.897405 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.897377 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b1ad4e-5bf585d59f-nlv5v"] Nov 03 04:23:08.986548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.986515 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" path="/var/lib/kubelet/pods/5ef707fe-4249-4ad5-a27f-265370cb9ef3/volumes" Nov 03 04:23:08.986965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:08.986934 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" path="/var/lib/kubelet/pods/952569b9-4a82-4650-82e7-79de667bbd6f/volumes" Nov 03 04:23:09.326368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:09.326330 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:09.326562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:09.326388 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:10.326678 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.326642 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:10.327352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.327302 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:10.546910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.546235 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 04:23:10.634201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.634112 5275 generic.go:356] "Generic (PLEG): container finished" podID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerID="d1909d9f29a0550e6993767d61a1697dac9e5906e6246486cf4684db72d36571" exitCode=0 Nov 03 04:23:10.634410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.634339 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" event={"ID":"e00de5aa-1c41-49e6-8422-0dacd79342ef","Type":"ContainerDied","Data":"d1909d9f29a0550e6993767d61a1697dac9e5906e6246486cf4684db72d36571"} Nov 03 04:23:10.636155 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.636125 5275 generic.go:356] "Generic (PLEG): container finished" podID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerID="2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5" exitCode=0 Nov 03 04:23:10.636366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.636187 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" Nov 03 04:23:10.636366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.636209 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" event={"ID":"eaa85c3b-27e9-4417-8c72-32cf5a365ad5","Type":"ContainerDied","Data":"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5"} Nov 03 04:23:10.636366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.636253 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56" event={"ID":"eaa85c3b-27e9-4417-8c72-32cf5a365ad5","Type":"ContainerDied","Data":"bc62d3b1651658a133a30418c30666f05c4762cd29c1a70a37f1410ecd008741"} Nov 03 04:23:10.636366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.636281 5275 scope.go:122] "RemoveContainer" containerID="2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5" Nov 03 04:23:10.651586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.651271 5275 scope.go:122] "RemoveContainer" containerID="2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5" Nov 03 04:23:10.651965 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:10.651863 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5\": container with ID starting with 2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5 not found: ID does not exist" containerID="2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5" Nov 03 04:23:10.651965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.651906 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5"} err="failed to get container status \"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5\": rpc error: code = NotFound desc = could not find container \"2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5\": container with ID starting with 2e8fe3cf7d03263ab5beb1815ee843ae076ecbc0c1a366bc084d692b0a2f1cd5 not found: ID does not exist" Nov 03 04:23:10.688893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.680538 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\" (UniqueName: \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\") pod \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " Nov 03 04:23:10.688893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.687042 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\") pod \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\" (UID: \"eaa85c3b-27e9-4417-8c72-32cf5a365ad5\") " Nov 03 04:23:10.688893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.687985 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume" pod "eaa85c3b-27e9-4417-8c72-32cf5a365ad5" (UID: "eaa85c3b-27e9-4417-8c72-32cf5a365ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:10.695410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.695373 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw" pod "eaa85c3b-27e9-4417-8c72-32cf5a365ad5" (UID: "eaa85c3b-27e9-4417-8c72-32cf5a365ad5"). InnerVolumeSpecName "kube-api-access-sxrrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:10.715125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.715009 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 04:23:10.788801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.788762 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\") pod \"e00de5aa-1c41-49e6-8422-0dacd79342ef\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " Nov 03 04:23:10.789008 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.788921 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\" (UniqueName: \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\") pod \"e00de5aa-1c41-49e6-8422-0dacd79342ef\" (UID: \"e00de5aa-1c41-49e6-8422-0dacd79342ef\") " Nov 03 04:23:10.789133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.789121 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrrw\" (UniqueName: \"kubernetes.io/projected/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-kube-api-access-sxrrw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:10.789174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.789135 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa85c3b-27e9-4417-8c72-32cf5a365ad5-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:10.789615 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.789590 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume" pod "e00de5aa-1c41-49e6-8422-0dacd79342ef" (UID: "e00de5aa-1c41-49e6-8422-0dacd79342ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:10.791431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.791409 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2" pod "e00de5aa-1c41-49e6-8422-0dacd79342ef" (UID: "e00de5aa-1c41-49e6-8422-0dacd79342ef"). InnerVolumeSpecName "kube-api-access-zwtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:10.890606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.890561 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtx2\" (UniqueName: \"kubernetes.io/projected/e00de5aa-1c41-49e6-8422-0dacd79342ef-kube-api-access-zwtx2\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:10.890606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:10.890607 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00de5aa-1c41-49e6-8422-0dacd79342ef-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:11.085118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.085013 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 04:23:11.095955 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.095918 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5a6246-76b99b5b69-ttd56"] Nov 03 04:23:11.325831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.325796 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:11.326059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.325888 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:11.671876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.671601 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" event={"ID":"e00de5aa-1c41-49e6-8422-0dacd79342ef","Type":"ContainerDied","Data":"42fc6c6fbdecdad1924e846dfd39df9e854320271bbaa17dce0b46112f083c20"} Nov 03 04:23:11.671876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.671674 5275 scope.go:122] "RemoveContainer" containerID="d1909d9f29a0550e6993767d61a1697dac9e5906e6246486cf4684db72d36571" Nov 03 04:23:11.671876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.671793 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq" Nov 03 04:23:11.851874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.851369 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 04:23:11.855459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:11.855418 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5a6246-7d57f5cd87-f2wdq"] Nov 03 04:23:12.325891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:12.325822 5275 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Nov 03 04:23:12.326088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:12.325912 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Nov 03 04:23:13.000292 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:13.000255 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" path="/var/lib/kubelet/pods/e00de5aa-1c41-49e6-8422-0dacd79342ef/volumes" Nov 03 04:23:13.000884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:13.000870 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" path="/var/lib/kubelet/pods/eaa85c3b-27e9-4417-8c72-32cf5a365ad5/volumes" Nov 03 04:23:13.336251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:13.336208 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:13.336526 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:13.336504 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:13.337659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:13.337637 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:15.048729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:15.048696 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4"] Nov 03 04:23:15.098374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:15.098341 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b804f98-mlgb4"] Nov 03 04:23:16.984245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:16.984207 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2b5472-a3d6-4842-8c26-2ed87140bdff" path="/var/lib/kubelet/pods/8d2b5472-a3d6-4842-8c26-2ed87140bdff/volumes" Nov 03 04:23:17.127396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:17.127356 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 04:23:20.115539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.115501 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:20.116027 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116013 5275 cpu_manager.go:414] "containerMap: removing container" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" Nov 03 04:23:20.116084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116028 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" Nov 03 04:23:20.116084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116048 5275 cpu_manager.go:414] "containerMap: removing container" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" Nov 03 04:23:20.116084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116053 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" Nov 03 04:23:20.116084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116075 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" Nov 03 04:23:20.116084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116080 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116087 5275 cpu_manager.go:414] "containerMap: removing container" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116092 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116149 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa85c3b-27e9-4417-8c72-32cf5a365ad5" containerName="payload-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116162 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00de5aa-1c41-49e6-8422-0dacd79342ef" containerName="payload-pre-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116170 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="952569b9-4a82-4650-82e7-79de667bbd6f" containerName="payload-pre-processing" Nov 03 04:23:20.116250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.116179 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef707fe-4249-4ad5-a27f-265370cb9ef3" containerName="payload-processing" Nov 03 04:23:20.121041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.121017 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.173137 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.173106 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\"" Nov 03 04:23:20.216855 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.216799 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.216869 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.216985 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217028 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzhj\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217088 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217230 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217214 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217267 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217341 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.217420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.217400 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.226566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.226531 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:20.319007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.318972 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319022 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319156 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319247 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319280 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319348 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319394 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzhj\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.319517 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319436 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319516 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319520 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319669 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.319981 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.320413 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.320459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.320442 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.322574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.322548 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.322802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.322772 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.381245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.381162 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.381245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.381166 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzhj\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.435594 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.435557 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:20.878522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.878491 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:20.881348 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:23:20.881319 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba9352_25e8_4cff_8173_6e63b324addc.slice/crio-c9cb2bb83ed93150582b58ad20bca9783e501d2dcf7491640ae20ded1c19b3ca WatchSource:0}: Error finding container c9cb2bb83ed93150582b58ad20bca9783e501d2dcf7491640ae20ded1c19b3ca: Status 404 returned error can't find the container with id c9cb2bb83ed93150582b58ad20bca9783e501d2dcf7491640ae20ded1c19b3ca Nov 03 04:23:20.884176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.884144 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:20.884284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.884210 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:20.884284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:20.884238 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:23:21.786874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:21.768406 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" event={"ID":"66ba9352-25e8-4cff-8173-6e63b324addc","Type":"ContainerStarted","Data":"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878"} Nov 03 04:23:21.786874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:21.768511 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" event={"ID":"66ba9352-25e8-4cff-8173-6e63b324addc","Type":"ContainerStarted","Data":"c9cb2bb83ed93150582b58ad20bca9783e501d2dcf7491640ae20ded1c19b3ca"} Nov 03 04:23:21.890308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:21.889883 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podStartSLOduration=1.8898645200000002 podStartE2EDuration="1.88986452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:23:21.889437484 +0000 UTC m=+4893.042125906" watchObservedRunningTime="2026-11-03 04:23:21.88986452 +0000 UTC m=+4893.042552939" Nov 03 04:23:22.128009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:22.127895 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" containerID="cri-o://5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f" gracePeriod=25 Nov 03 04:23:22.436728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:22.436648 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:22.438189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:22.438160 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:22.438266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:22.438207 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:23.436080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:23.436044 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:23.436492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:23.436107 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:24.436449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:24.436412 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:24.436876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:24.436474 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:25.436038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:25.436005 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:25.436219 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:25.436061 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:25.948519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:25.948482 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:23:26.436228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:26.436188 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:26.436427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:26.436259 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:27.436425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:27.436387 5275 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body= Nov 03 04:23:27.436919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:27.436443 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" Nov 03 04:23:28.627310 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:28.627271 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:28.627812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:28.627673 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:28.628726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:28.628701 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:31.874234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:31.874196 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:31.874694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:31.874448 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" containerID="cri-o://5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878" gracePeriod=30 Nov 03 04:23:32.336084 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:32.336009 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:23:35.948506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:35.948469 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:23:37.141633 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.141573 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:37.240306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240270 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240310 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240339 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240390 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240453 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240472 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240506 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240604 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240602 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:37.240937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240680 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\") pod \"66ba9352-25e8-4cff-8173-6e63b324addc\" (UID: \"66ba9352-25e8-4cff-8173-6e63b324addc\") " Nov 03 04:23:37.240937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240673 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:37.241102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240929 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:37.241102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.240774 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:37.241102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241056 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:37.241210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241162 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.241210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241177 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.241210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241186 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba9352-25e8-4cff-8173-6e63b324addc-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.241210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241195 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.241210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.241203 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.243592 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.243560 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:23:37.243592 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.243571 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:37.243766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.243629 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:37.243766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.243649 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj" pod "66ba9352-25e8-4cff-8173-6e63b324addc" (UID: "66ba9352-25e8-4cff-8173-6e63b324addc"). InnerVolumeSpecName "kube-api-access-ntzhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:37.342294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.342262 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba9352-25e8-4cff-8173-6e63b324addc-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.342294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.342292 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntzhj\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-kube-api-access-ntzhj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.342294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.342302 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba9352-25e8-4cff-8173-6e63b324addc-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.342576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.342310 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba9352-25e8-4cff-8173-6e63b324addc-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:37.887466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.887431 5275 generic.go:356] "Generic (PLEG): container finished" podID="66ba9352-25e8-4cff-8173-6e63b324addc" containerID="5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878" exitCode=0 Nov 03 04:23:37.887653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.887478 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" Nov 03 04:23:37.887653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.887503 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" event={"ID":"66ba9352-25e8-4cff-8173-6e63b324addc","Type":"ContainerDied","Data":"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878"} Nov 03 04:23:37.887653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.887558 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd" event={"ID":"66ba9352-25e8-4cff-8173-6e63b324addc","Type":"ContainerDied","Data":"c9cb2bb83ed93150582b58ad20bca9783e501d2dcf7491640ae20ded1c19b3ca"} Nov 03 04:23:37.887653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.887581 5275 scope.go:122] "RemoveContainer" containerID="5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878" Nov 03 04:23:37.902047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.902029 5275 scope.go:122] "RemoveContainer" containerID="5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878" Nov 03 04:23:37.902318 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:37.902299 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878\": container with ID starting with 5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878 not found: ID does not exist" containerID="5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878" Nov 03 04:23:37.902370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:37.902330 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878"} err="failed to get container status \"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878\": rpc error: code = NotFound desc = could not find container \"5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878\": container with ID starting with 5df7c3e15b678a3f41885676d812561c2ae1a7e43fb504e98e1335b984b18878 not found: ID does not exist" Nov 03 04:23:38.104382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:38.104349 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:38.156282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:38.156205 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-f5wbd"] Nov 03 04:23:38.983642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:38.983612 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" path="/var/lib/kubelet/pods/66ba9352-25e8-4cff-8173-6e63b324addc/volumes" Nov 03 04:23:42.382236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.382209 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 04:23:42.503913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.503792 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\") pod \"accbe785-7658-4ba4-b61c-7250e0c1878e\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " Nov 03 04:23:42.504098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.503975 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\" (UniqueName: \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\") pod \"accbe785-7658-4ba4-b61c-7250e0c1878e\" (UID: \"accbe785-7658-4ba4-b61c-7250e0c1878e\") " Nov 03 04:23:42.504274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.504250 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume" pod "accbe785-7658-4ba4-b61c-7250e0c1878e" (UID: "accbe785-7658-4ba4-b61c-7250e0c1878e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:42.504463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.504427 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accbe785-7658-4ba4-b61c-7250e0c1878e-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:42.506383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.506357 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59" pod "accbe785-7658-4ba4-b61c-7250e0c1878e" (UID: "accbe785-7658-4ba4-b61c-7250e0c1878e"). InnerVolumeSpecName "kube-api-access-j4g59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:42.605494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.605457 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4g59\" (UniqueName: \"kubernetes.io/projected/accbe785-7658-4ba4-b61c-7250e0c1878e-kube-api-access-j4g59\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:42.927111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.927073 5275 generic.go:356] "Generic (PLEG): container finished" podID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerID="5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f" exitCode=0 Nov 03 04:23:42.927312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.927161 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" Nov 03 04:23:42.927312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.927174 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" event={"ID":"accbe785-7658-4ba4-b61c-7250e0c1878e","Type":"ContainerDied","Data":"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f"} Nov 03 04:23:42.927312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.927276 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j" event={"ID":"accbe785-7658-4ba4-b61c-7250e0c1878e","Type":"ContainerDied","Data":"dbee691672fa848c030a325f265a4fed71fb68e375cb75119b3c0e977d649734"} Nov 03 04:23:42.927312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.927299 5275 scope.go:122] "RemoveContainer" containerID="5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f" Nov 03 04:23:42.937782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.937764 5275 scope.go:122] "RemoveContainer" containerID="5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f" Nov 03 04:23:42.938176 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:42.938154 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f\": container with ID starting with 5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f not found: ID does not exist" containerID="5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f" Nov 03 04:23:42.938303 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:42.938188 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f"} err="failed to get container status \"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f\": rpc error: code = NotFound desc = could not find container \"5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f\": container with ID starting with 5c5ae09a36b2b1987b1de71e52275402525d22770093e8446cc24f098d80c98f not found: ID does not exist" Nov 03 04:23:43.093691 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:43.093656 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 04:23:43.144466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:43.144425 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-efa9bf-76b5cbdcff-nlm8j"] Nov 03 04:23:43.546950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:43.546917 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:43.547379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:43.547202 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" containerID="cri-o://fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403" gracePeriod=30 Nov 03 04:23:44.984128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:44.984097 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" path="/var/lib/kubelet/pods/accbe785-7658-4ba4-b61c-7250e0c1878e/volumes" Nov 03 04:23:48.846555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.846528 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:48.969307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.969275 5275 generic.go:356] "Generic (PLEG): container finished" podID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerID="fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403" exitCode=0 Nov 03 04:23:48.969476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.969341 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" Nov 03 04:23:48.969476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.969375 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" event={"ID":"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9","Type":"ContainerDied","Data":"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403"} Nov 03 04:23:48.969573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.969477 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn" event={"ID":"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9","Type":"ContainerDied","Data":"5dd05b2f371d7814cddeedb0a9f07d7d7bca281de7f2123b18e740961ec18018"} Nov 03 04:23:48.969573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.969511 5275 scope.go:122] "RemoveContainer" containerID="fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403" Nov 03 04:23:48.981335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.981309 5275 scope.go:122] "RemoveContainer" containerID="fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403" Nov 03 04:23:48.981639 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:23:48.981619 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403\": container with ID starting with fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403 not found: ID does not exist" containerID="fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403" Nov 03 04:23:48.981730 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:48.981652 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403"} err="failed to get container status \"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403\": rpc error: code = NotFound desc = could not find container \"fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403\": container with ID starting with fe8252be49326cb4a11b128b345e912cb7f2c2240b5c379fca47776a0054a403 not found: ID does not exist" Nov 03 04:23:49.018756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.018717 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.018968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.018799 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.018968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.018891 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.018968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.018939 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.018977 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019025 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019054 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019084 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019087 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.019453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019206 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\") pod \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\" (UID: \"b3d071ba-c366-4047-bbe1-c4d4af7bc7c9\") " Nov 03 04:23:49.019514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019448 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:49.019944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019697 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.019944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019904 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.019944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019926 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.019944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.019942 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.020647 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.020603 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.021509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.021469 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.022901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.022816 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:23:49.022994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.022907 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "kube-api-access-l8ndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:49.023062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.022994 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:49.023810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.023775 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy" pod "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" (UID: "b3d071ba-c366-4047-bbe1-c4d4af7bc7c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120915 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ndw\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-kube-api-access-l8ndw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120946 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120955 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120963 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120973 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.121000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.120983 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.395939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.393368 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:49.407349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:49.407319 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9tmdn"] Nov 03 04:23:50.984216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:50.984183 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" path="/var/lib/kubelet/pods/b3d071ba-c366-4047-bbe1-c4d4af7bc7c9/volumes" Nov 03 04:23:54.490308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:23:54.490273 5275 scope.go:122] "RemoveContainer" containerID="43ff70a08bb5688bd46fab0c5decd2ef5392ff47c5cf50f5d32f947930f007e8" Nov 03 04:24:02.337281 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:24:02.337237 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:24:08.020894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:08.020841 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:24:08.021430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:08.021135 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" containerID="cri-o://ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5" gracePeriod=30 Nov 03 04:24:10.644792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.644749 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt"] Nov 03 04:24:10.645263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645246 5275 cpu_manager.go:414] "containerMap: removing container" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" Nov 03 04:24:10.645315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645266 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" Nov 03 04:24:10.645315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645301 5275 cpu_manager.go:414] "containerMap: removing container" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" Nov 03 04:24:10.645315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645307 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" Nov 03 04:24:10.645427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645320 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" Nov 03 04:24:10.645427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645327 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" Nov 03 04:24:10.645427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645400 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba9352-25e8-4cff-8173-6e63b324addc" containerName="istio-proxy" Nov 03 04:24:10.645427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645416 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="accbe785-7658-4ba4-b61c-7250e0c1878e" containerName="payload-pre-processing" Nov 03 04:24:10.645427 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.645425 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d071ba-c366-4047-bbe1-c4d4af7bc7c9" containerName="istio-proxy" Nov 03 04:24:10.650448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.650425 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.752161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.752126 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:24:10.752161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.752135 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:24:10.752161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.752135 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:24:10.752511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.752271 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:24:10.754736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.754711 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt"] Nov 03 04:24:10.761859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.761797 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.762046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.762023 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.762168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.762153 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xj5\" (UniqueName: \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.863095 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.863060 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.863280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.863154 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xj5\" (UniqueName: \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.863280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.863215 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.864075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.864041 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.865548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.865523 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.924360 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.924276 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xj5\" (UniqueName: \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\") pod \"maas-api-revoke-sub-23dc2b5e-8cxxt\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:10.966896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:10.966827 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:11.345373 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:11.345348 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt"] Nov 03 04:24:11.347282 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:24:11.347256 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dad3d6_bf07_4181_b669_43b687c345e8.slice/crio-383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8 WatchSource:0}: Error finding container 383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8: Status 404 returned error can't find the container with id 383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8 Nov 03 04:24:12.124899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:12.124865 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" event={"ID":"59dad3d6-bf07-4181-b669-43b687c345e8","Type":"ContainerStarted","Data":"9d58cabfebfc6ef2e60e45d39e9e33eb672e2880e77c256f052575b7869891a8"} Nov 03 04:24:12.125310 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:12.124920 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" event={"ID":"59dad3d6-bf07-4181-b669-43b687c345e8","Type":"ContainerStarted","Data":"383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8"} Nov 03 04:24:12.240835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:12.240782 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" podStartSLOduration=2.2407658599999998 podStartE2EDuration="2.24076586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:24:12.238877087 +0000 UTC m=+4943.391565506" watchObservedRunningTime="2026-11-03 04:24:12.24076586 +0000 UTC m=+4943.393454281" Nov 03 04:24:12.696879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:12.696806 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xq4ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:24:12.697066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:12.696913 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:24:13.133307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.133271 5275 generic.go:356] "Generic (PLEG): container finished" podID="59dad3d6-bf07-4181-b669-43b687c345e8" containerID="9d58cabfebfc6ef2e60e45d39e9e33eb672e2880e77c256f052575b7869891a8" exitCode=0 Nov 03 04:24:13.133681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.133358 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" event={"ID":"59dad3d6-bf07-4181-b669-43b687c345e8","Type":"ContainerDied","Data":"9d58cabfebfc6ef2e60e45d39e9e33eb672e2880e77c256f052575b7869891a8"} Nov 03 04:24:13.283369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.283343 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:24:13.388971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.388837 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.388971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.388921 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.388971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.388944 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389038 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389058 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389085 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389122 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389163 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389229 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\") pod \"adb2ca5e-d035-4d57-840c-13686d47cdb8\" (UID: \"adb2ca5e-d035-4d57-840c-13686d47cdb8\") " Nov 03 04:24:13.389621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389279 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.389621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389323 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:13.389621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389501 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.389621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389543 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.389621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389563 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.390098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389765 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.390098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389785 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.390098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389799 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.390098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389816 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2ca5e-d035-4d57-840c-13686d47cdb8-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.390098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.389945 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.391666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.391600 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "kube-api-access-82cfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:13.391666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.391623 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:13.392135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.392107 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:24:13.392234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.392163 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy" pod "adb2ca5e-d035-4d57-840c-13686d47cdb8" (UID: "adb2ca5e-d035-4d57-840c-13686d47cdb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.491393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.491359 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.491393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.491391 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.491393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.491402 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82cfb\" (UniqueName: \"kubernetes.io/projected/adb2ca5e-d035-4d57-840c-13686d47cdb8-kube-api-access-82cfb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.491664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:13.491413 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2ca5e-d035-4d57-840c-13686d47cdb8-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:14.141714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.141677 5275 generic.go:356] "Generic (PLEG): container finished" podID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerID="ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5" exitCode=0 Nov 03 04:24:14.142220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.141731 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" Nov 03 04:24:14.142220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.141741 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" event={"ID":"adb2ca5e-d035-4d57-840c-13686d47cdb8","Type":"ContainerDied","Data":"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5"} Nov 03 04:24:14.142220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.141798 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld" event={"ID":"adb2ca5e-d035-4d57-840c-13686d47cdb8","Type":"ContainerDied","Data":"c522745c44053c6c78afc61c470026794aac0a8d88506181e4a0e91c97d22d58"} Nov 03 04:24:14.142220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.141825 5275 scope.go:122] "RemoveContainer" containerID="ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5" Nov 03 04:24:14.153995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.153969 5275 scope.go:122] "RemoveContainer" containerID="ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5" Nov 03 04:24:14.154361 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:24:14.154340 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5\": container with ID starting with ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5 not found: ID does not exist" containerID="ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5" Nov 03 04:24:14.154421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.154372 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5"} err="failed to get container status \"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5\": rpc error: code = NotFound desc = could not find container \"ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5\": container with ID starting with ac6bc3bd4868958d3ad0ab6d148c9e38a2f5ab07995115f29206368f866701a5 not found: ID does not exist" Nov 03 04:24:14.285167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.285134 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:14.309020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.308978 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:24:14.358351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.358310 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xq4ld"] Nov 03 04:24:14.401465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.401371 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\") pod \"59dad3d6-bf07-4181-b669-43b687c345e8\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " Nov 03 04:24:14.401672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.401476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\") pod \"59dad3d6-bf07-4181-b669-43b687c345e8\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " Nov 03 04:24:14.401672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.401531 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\" (UniqueName: \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\") pod \"59dad3d6-bf07-4181-b669-43b687c345e8\" (UID: \"59dad3d6-bf07-4181-b669-43b687c345e8\") " Nov 03 04:24:14.401889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.401862 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca" pod "59dad3d6-bf07-4181-b669-43b687c345e8" (UID: "59dad3d6-bf07-4181-b669-43b687c345e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:14.403786 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.403754 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5" pod "59dad3d6-bf07-4181-b669-43b687c345e8" (UID: "59dad3d6-bf07-4181-b669-43b687c345e8"). InnerVolumeSpecName "kube-api-access-w6xj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:14.403958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.403783 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca" pod "59dad3d6-bf07-4181-b669-43b687c345e8" (UID: "59dad3d6-bf07-4181-b669-43b687c345e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:14.503023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.502974 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:14.503023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.503020 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6xj5\" (UniqueName: \"kubernetes.io/projected/59dad3d6-bf07-4181-b669-43b687c345e8-kube-api-access-w6xj5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:14.503023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.503032 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dad3d6-bf07-4181-b669-43b687c345e8-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:14.983794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:14.983754 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" path="/var/lib/kubelet/pods/adb2ca5e-d035-4d57-840c-13686d47cdb8/volumes" Nov 03 04:24:15.150938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:15.150906 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" Nov 03 04:24:15.151445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:15.150912 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt" event={"ID":"59dad3d6-bf07-4181-b669-43b687c345e8","Type":"ContainerDied","Data":"383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8"} Nov 03 04:24:15.151445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:15.151032 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383c6425e3cb31e8cf5927fb6b7984dca1bc9020896c54fe3752a73c999816f8" Nov 03 04:24:21.358130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358084 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z"] Nov 03 04:24:21.358585 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358569 5275 cpu_manager.go:414] "containerMap: removing container" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" Nov 03 04:24:21.358642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358587 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" Nov 03 04:24:21.358642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358614 5275 cpu_manager.go:414] "containerMap: removing container" podUID="59dad3d6-bf07-4181-b669-43b687c345e8" containerName="revoke-keys" Nov 03 04:24:21.358642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358619 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dad3d6-bf07-4181-b669-43b687c345e8" containerName="revoke-keys" Nov 03 04:24:21.358744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358683 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb2ca5e-d035-4d57-840c-13686d47cdb8" containerName="istio-proxy" Nov 03 04:24:21.358744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.358695 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dad3d6-bf07-4181-b669-43b687c345e8" containerName="revoke-keys" Nov 03 04:24:21.362334 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.362317 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.421123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.421078 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z"] Nov 03 04:24:21.466045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.466003 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:24:21.466045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.466005 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:24:21.466286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.466126 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:24:21.466286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.466173 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:24:21.479924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.479881 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.480113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.480074 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.480173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.480137 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8zc\" (UniqueName: \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.581001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.580961 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.581232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.581047 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8zc\" (UniqueName: \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.581232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.581154 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.581813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.581782 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.583561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.583538 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.642129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.642051 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8zc\" (UniqueName: \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\") pod \"maas-api-revoke-sub-b0741006-5zv2z\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:21.676176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:21.676148 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:22.059520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:22.059490 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z"] Nov 03 04:24:22.061359 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:24:22.061311 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c41fb6_f877_445f_a704_8217d002ef86.slice/crio-427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997 WatchSource:0}: Error finding container 427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997: Status 404 returned error can't find the container with id 427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997 Nov 03 04:24:22.204914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:22.204874 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" event={"ID":"76c41fb6-f877-445f-a704-8217d002ef86","Type":"ContainerStarted","Data":"427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997"} Nov 03 04:24:23.213366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:23.213275 5275 generic.go:356] "Generic (PLEG): container finished" podID="76c41fb6-f877-445f-a704-8217d002ef86" containerID="ef97b2a3e7a592bf02285c4688f751d61e5406d6392a06da36951ff68268033a" exitCode=0 Nov 03 04:24:23.213796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:23.213365 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" event={"ID":"76c41fb6-f877-445f-a704-8217d002ef86","Type":"ContainerDied","Data":"ef97b2a3e7a592bf02285c4688f751d61e5406d6392a06da36951ff68268033a"} Nov 03 04:24:24.362958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.362932 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:24.511329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.511223 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\" (UniqueName: \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\") pod \"76c41fb6-f877-445f-a704-8217d002ef86\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " Nov 03 04:24:24.511532 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.511333 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\") pod \"76c41fb6-f877-445f-a704-8217d002ef86\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " Nov 03 04:24:24.511532 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.511377 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\") pod \"76c41fb6-f877-445f-a704-8217d002ef86\" (UID: \"76c41fb6-f877-445f-a704-8217d002ef86\") " Nov 03 04:24:24.511809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.511780 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca" pod "76c41fb6-f877-445f-a704-8217d002ef86" (UID: "76c41fb6-f877-445f-a704-8217d002ef86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:24.511987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.511968 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:24.513744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.513719 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc" pod "76c41fb6-f877-445f-a704-8217d002ef86" (UID: "76c41fb6-f877-445f-a704-8217d002ef86"). InnerVolumeSpecName "kube-api-access-gb8zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:24.513930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.513827 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca" pod "76c41fb6-f877-445f-a704-8217d002ef86" (UID: "76c41fb6-f877-445f-a704-8217d002ef86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:24.613464 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.613423 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c41fb6-f877-445f-a704-8217d002ef86-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:24.613464 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:24.613462 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8zc\" (UniqueName: \"kubernetes.io/projected/76c41fb6-f877-445f-a704-8217d002ef86-kube-api-access-gb8zc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:25.228040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:25.228011 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" Nov 03 04:24:25.228040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:25.228031 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z" event={"ID":"76c41fb6-f877-445f-a704-8217d002ef86","Type":"ContainerDied","Data":"427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997"} Nov 03 04:24:25.228273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:25.228089 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427478d80c689638c84e34363afaad78dab8ac1978ddf1c5b574554b9545b997" Nov 03 04:24:29.400588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:29.400544 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 04:24:29.401174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:29.400816 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" containerID="cri-o://a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501" gracePeriod=30 Nov 03 04:24:30.509627 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.509578 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt"] Nov 03 04:24:30.510253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.510229 5275 cpu_manager.go:414] "containerMap: removing container" podUID="76c41fb6-f877-445f-a704-8217d002ef86" containerName="revoke-keys" Nov 03 04:24:30.510344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.510255 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c41fb6-f877-445f-a704-8217d002ef86" containerName="revoke-keys" Nov 03 04:24:30.510430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.510417 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c41fb6-f877-445f-a704-8217d002ef86" containerName="revoke-keys" Nov 03 04:24:30.516172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.516135 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.571919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.571882 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt"] Nov 03 04:24:30.580497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.580462 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.580661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.580602 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.580740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.580719 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpw6p\" (UniqueName: \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.617538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.617505 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:24:30.617538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.617508 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:24:30.617745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.617540 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:24:30.618758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.618739 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:24:30.682348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.682311 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.682537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.682372 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpw6p\" (UniqueName: \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.682537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.682462 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.683099 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.683072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.684826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.684805 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.743013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.742959 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpw6p\" (UniqueName: \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\") pod \"maas-api-revoke-sub-358da859-669vt\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:30.778264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.778227 5275 patch_prober.go:29] interesting pod/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:24:30.778459 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.778288 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:24:30.831640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:30.831614 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:31.205123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:31.205097 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt"] Nov 03 04:24:31.207444 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:24:31.207414 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e924eba_2a2a_4829_a342_74262e1a2bf1.slice/crio-a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735 WatchSource:0}: Error finding container a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735: Status 404 returned error can't find the container with id a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735 Nov 03 04:24:31.265801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:31.265765 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" event={"ID":"0e924eba-2a2a-4829-a342-74262e1a2bf1","Type":"ContainerStarted","Data":"a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735"} Nov 03 04:24:32.274183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:32.274146 5275 generic.go:356] "Generic (PLEG): container finished" podID="0e924eba-2a2a-4829-a342-74262e1a2bf1" containerID="f16cd96736761c8cf35ca5a14c919f5a4cf2bcc20a737a7f402bcb8a494d56e1" exitCode=0 Nov 03 04:24:32.274636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:32.274230 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" event={"ID":"0e924eba-2a2a-4829-a342-74262e1a2bf1","Type":"ContainerDied","Data":"f16cd96736761c8cf35ca5a14c919f5a4cf2bcc20a737a7f402bcb8a494d56e1"} Nov 03 04:24:32.330447 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:24:32.330413 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:24:33.449090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.449065 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:33.517507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.517476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\") pod \"0e924eba-2a2a-4829-a342-74262e1a2bf1\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " Nov 03 04:24:33.517685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.517549 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\") pod \"0e924eba-2a2a-4829-a342-74262e1a2bf1\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " Nov 03 04:24:33.517726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.517702 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\" (UniqueName: \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\") pod \"0e924eba-2a2a-4829-a342-74262e1a2bf1\" (UID: \"0e924eba-2a2a-4829-a342-74262e1a2bf1\") " Nov 03 04:24:33.517905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.517880 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca" pod "0e924eba-2a2a-4829-a342-74262e1a2bf1" (UID: "0e924eba-2a2a-4829-a342-74262e1a2bf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:33.518069 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.518051 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:33.519686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.519663 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca" pod "0e924eba-2a2a-4829-a342-74262e1a2bf1" (UID: "0e924eba-2a2a-4829-a342-74262e1a2bf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:33.519838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.519816 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p" pod "0e924eba-2a2a-4829-a342-74262e1a2bf1" (UID: "0e924eba-2a2a-4829-a342-74262e1a2bf1"). InnerVolumeSpecName "kube-api-access-bpw6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:33.619709 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.619623 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpw6p\" (UniqueName: \"kubernetes.io/projected/0e924eba-2a2a-4829-a342-74262e1a2bf1-kube-api-access-bpw6p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:33.619709 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:33.619655 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e924eba-2a2a-4829-a342-74262e1a2bf1-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.292247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.292215 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" Nov 03 04:24:34.292247 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.292231 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt" event={"ID":"0e924eba-2a2a-4829-a342-74262e1a2bf1","Type":"ContainerDied","Data":"a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735"} Nov 03 04:24:34.292531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.292290 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d6d2df498e1d110b66cf792c37c99df00da472fdd7e831f71db6524b069735" Nov 03 04:24:34.667272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.667248 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 04:24:34.730149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730111 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730234 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730273 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730293 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730346 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730367 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730388 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730430 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730462 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\") pod \"ba1ea932-483c-42fe-962b-490e4e43f2b6\" (UID: \"ba1ea932-483c-42fe-962b-490e4e43f2b6\") " Nov 03 04:24:34.730818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730611 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:34.730818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730597 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:34.730818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730663 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:34.730818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730675 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:34.730818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730815 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.731098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730825 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:34.731098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730833 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1ea932-483c-42fe-962b-490e4e43f2b6-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.731098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730897 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.731098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.730911 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.732751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.732723 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:34.732910 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.732797 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:24:34.733002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.732969 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:34.733002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.732980 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc" pod "ba1ea932-483c-42fe-962b-490e4e43f2b6" (UID: "ba1ea932-483c-42fe-962b-490e4e43f2b6"). InnerVolumeSpecName "kube-api-access-wjwnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:34.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.831681 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.831718 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.831731 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1ea932-483c-42fe-962b-490e4e43f2b6-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.831751 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwnc\" (UniqueName: \"kubernetes.io/projected/ba1ea932-483c-42fe-962b-490e4e43f2b6-kube-api-access-wjwnc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:34.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:34.831765 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1ea932-483c-42fe-962b-490e4e43f2b6-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:35.299905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.299873 5275 generic.go:356] "Generic (PLEG): container finished" podID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerID="a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501" exitCode=0 Nov 03 04:24:35.300059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.299950 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerDied","Data":"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501"} Nov 03 04:24:35.300059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.300004 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" Nov 03 04:24:35.300059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.300033 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md" event={"ID":"ba1ea932-483c-42fe-962b-490e4e43f2b6","Type":"ContainerDied","Data":"50f75164beef9435260794b830075730fa62911500dd6b5bae986b5bccf3de45"} Nov 03 04:24:35.300190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.300064 5275 scope.go:122] "RemoveContainer" containerID="a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501" Nov 03 04:24:35.310334 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.310312 5275 scope.go:122] "RemoveContainer" containerID="9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f" Nov 03 04:24:35.321339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.321318 5275 scope.go:122] "RemoveContainer" containerID="a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501" Nov 03 04:24:35.321655 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:24:35.321637 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501\": container with ID starting with a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501 not found: ID does not exist" containerID="a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501" Nov 03 04:24:35.321727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.321664 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501"} err="failed to get container status \"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501\": rpc error: code = NotFound desc = could not find container \"a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501\": container with ID starting with a8dcdc655dc609a73ca9eecb8d82660de5e7050f0bd97e6a127d3b152aa9c501 not found: ID does not exist" Nov 03 04:24:35.321727 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.321679 5275 scope.go:122] "RemoveContainer" containerID="9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f" Nov 03 04:24:35.321942 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:24:35.321921 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f\": container with ID starting with 9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f not found: ID does not exist" containerID="9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f" Nov 03 04:24:35.322004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.321951 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f"} err="failed to get container status \"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f\": rpc error: code = NotFound desc = could not find container \"9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f\": container with ID starting with 9a16447528bacb76b8d2668a13c0280d73bb14c17a23376f5687aa112d85308f not found: ID does not exist" Nov 03 04:24:35.517525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.517489 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 04:24:35.567715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:35.567634 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-286a8c-openshift-default-fcdff4c85-mm6md"] Nov 03 04:24:36.984653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:36.984617 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" path="/var/lib/kubelet/pods/ba1ea932-483c-42fe-962b-490e4e43f2b6/volumes" Nov 03 04:24:46.737723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.737684 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld"] Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738141 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0e924eba-2a2a-4829-a342-74262e1a2bf1" containerName="revoke-keys" Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738155 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e924eba-2a2a-4829-a342-74262e1a2bf1" containerName="revoke-keys" Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738180 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738185 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738211 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.738280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738216 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.738490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738298 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.738490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738304 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e924eba-2a2a-4829-a342-74262e1a2bf1" containerName="revoke-keys" Nov 03 04:24:46.738490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.738314 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1ea932-483c-42fe-962b-490e4e43f2b6" containerName="istio-proxy" Nov 03 04:24:46.743174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.743147 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.795577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.795543 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:24:46.795892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.795865 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:24:46.845058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.845016 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:24:46.845058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.845053 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:24:46.847763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.847729 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld"] Nov 03 04:24:46.870483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.870440 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkx7h\" (UniqueName: \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.870705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.870502 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.870705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.870668 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.972472 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.972422 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.972732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.972591 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkx7h\" (UniqueName: \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.972732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.972635 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.973484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.973456 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:46.975762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:46.975728 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:47.034151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:47.034123 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkx7h\" (UniqueName: \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\") pod \"maas-api-revoke-sub-4e44c907-c7mld\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:47.063798 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:47.063757 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:47.441292 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:47.441259 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld"] Nov 03 04:24:47.443090 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:24:47.443059 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bea7ac5_92cb_4665_be7c_95ec0b76c8e9.slice/crio-0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa WatchSource:0}: Error finding container 0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa: Status 404 returned error can't find the container with id 0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa Nov 03 04:24:48.393531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:48.393487 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" event={"ID":"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9","Type":"ContainerStarted","Data":"3c9b190f38dbdd2e63e80d404761529e2dc66a25118e8a9a2ede4e5418ec6c14"} Nov 03 04:24:48.394208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:48.394178 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" event={"ID":"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9","Type":"ContainerStarted","Data":"0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa"} Nov 03 04:24:48.510789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:48.510722 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" podStartSLOduration=2.5107072280000002 podStartE2EDuration="2.510707228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:24:48.509809832 +0000 UTC m=+4979.662498264" watchObservedRunningTime="2026-11-03 04:24:48.510707228 +0000 UTC m=+4979.663395705" Nov 03 04:24:49.401244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:49.401207 5275 generic.go:356] "Generic (PLEG): container finished" podID="6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" containerID="3c9b190f38dbdd2e63e80d404761529e2dc66a25118e8a9a2ede4e5418ec6c14" exitCode=0 Nov 03 04:24:49.401677 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:49.401290 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" event={"ID":"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9","Type":"ContainerDied","Data":"3c9b190f38dbdd2e63e80d404761529e2dc66a25118e8a9a2ede4e5418ec6c14"} Nov 03 04:24:50.595800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.595772 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:50.623994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.623955 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\" (UniqueName: \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\") pod \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " Nov 03 04:24:50.624161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.624061 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\") pod \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " Nov 03 04:24:50.624161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.624139 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\") pod \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\" (UID: \"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9\") " Nov 03 04:24:50.624467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.624436 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca" pod "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" (UID: "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:50.626404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.626377 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca" pod "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" (UID: "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:50.626503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.626385 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h" pod "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" (UID: "6bea7ac5-92cb-4665-be7c-95ec0b76c8e9"). InnerVolumeSpecName "kube-api-access-rkx7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:50.725243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.725166 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.725243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.725196 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkx7h\" (UniqueName: \"kubernetes.io/projected/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-kube-api-access-rkx7h\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.725243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:50.725207 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:51.414539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:51.414510 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" Nov 03 04:24:51.414729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:51.414534 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld" event={"ID":"6bea7ac5-92cb-4665-be7c-95ec0b76c8e9","Type":"ContainerDied","Data":"0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa"} Nov 03 04:24:51.414729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:51.414595 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3ce3250acfd02cecf078acb676bb271609d9822fc0259987f256f359e5a2fa" Nov 03 04:24:54.547720 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:54.547685 5275 scope.go:122] "RemoveContainer" containerID="c1d47631cb362dbb532b836b82080eb11261bdd5c98c5973d3086e0b63dd723d" Nov 03 04:24:54.557809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:54.557781 5275 scope.go:122] "RemoveContainer" containerID="b8a237660d4ecccf01374b82fe32d24977b0662e658d1aa65c1d5cf02721aa88" Nov 03 04:24:59.566697 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:24:59.566651 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:25:02.330882 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:02.330832 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:25:05.902833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:05.902794 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 04:25:10.903629 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:10.903560 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" containerID="cri-o://9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a" gracePeriod=25 Nov 03 04:25:14.587330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:14.587263 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" podUID="3c7d5936-3b42-4299-bef7-050cee937bfb" containerName="main" containerID="cri-o://76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25" gracePeriod=45 Nov 03 04:25:14.854348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:14.854276 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:25:15.020248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020210 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020283 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020320 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020440 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\" (UniqueName: \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020473 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020523 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\") pod \"3c7d5936-3b42-4299-bef7-050cee937bfb\" (UID: \"3c7d5936-3b42-4299-bef7-050cee937bfb\") " Nov 03 04:25:15.020561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020542 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:15.020791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020577 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:15.020876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020815 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:15.020928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020893 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.020997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.020935 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.022722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.022696 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:15.022904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.022881 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:15.023124 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.023107 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf" pod "3c7d5936-3b42-4299-bef7-050cee937bfb" (UID: "3c7d5936-3b42-4299-bef7-050cee937bfb"). InnerVolumeSpecName "kube-api-access-tjftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:15.121962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.121878 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjftf\" (UniqueName: \"kubernetes.io/projected/3c7d5936-3b42-4299-bef7-050cee937bfb-kube-api-access-tjftf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.121962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.121913 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7d5936-3b42-4299-bef7-050cee937bfb-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.121962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.121924 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.121962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.121934 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7d5936-3b42-4299-bef7-050cee937bfb-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:15.581552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.581519 5275 generic.go:356] "Generic (PLEG): container finished" podID="3c7d5936-3b42-4299-bef7-050cee937bfb" containerID="76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25" exitCode=0 Nov 03 04:25:15.581733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.581587 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" event={"ID":"3c7d5936-3b42-4299-bef7-050cee937bfb","Type":"ContainerDied","Data":"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25"} Nov 03 04:25:15.581733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.581644 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" event={"ID":"3c7d5936-3b42-4299-bef7-050cee937bfb","Type":"ContainerDied","Data":"66979ad81e07c57c610cf76e401e9ca5704874e0b21800863c5401578dd710e6"} Nov 03 04:25:15.581733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.581654 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd" Nov 03 04:25:15.581733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.581665 5275 scope.go:122] "RemoveContainer" containerID="76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25" Nov 03 04:25:15.592351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.592190 5275 scope.go:122] "RemoveContainer" containerID="76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25" Nov 03 04:25:15.592610 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:15.592493 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25\": container with ID starting with 76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25 not found: ID does not exist" containerID="76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25" Nov 03 04:25:15.592610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.592523 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25"} err="failed to get container status \"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25\": rpc error: code = NotFound desc = could not find container \"76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25\": container with ID starting with 76060936b799c85f2aa4b8faad71877d3b72effddc7ea2fa7a062e07bb291f25 not found: ID does not exist" Nov 03 04:25:15.745721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.745684 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:25:15.749110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:15.749083 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3d8dc5/rate-test-model-3d8dc5-kserve-595bcfdf5c-m9kvd"] Nov 03 04:25:16.568836 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.568793 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:16.569303 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569286 5275 cpu_manager.go:414] "containerMap: removing container" podUID="6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" containerName="revoke-keys" Nov 03 04:25:16.569407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569304 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" containerName="revoke-keys" Nov 03 04:25:16.569407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569317 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3c7d5936-3b42-4299-bef7-050cee937bfb" containerName="main" Nov 03 04:25:16.569407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569322 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7d5936-3b42-4299-bef7-050cee937bfb" containerName="main" Nov 03 04:25:16.569407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569407 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7d5936-3b42-4299-bef7-050cee937bfb" containerName="main" Nov 03 04:25:16.569632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.569424 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" containerName="revoke-keys" Nov 03 04:25:16.574282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.574258 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.626680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.626637 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\"" Nov 03 04:25:16.680255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.680218 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:16.738771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738732 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.738771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738781 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738821 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738865 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znvq\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738892 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738927 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.738983 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.739011 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.739083 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.739059 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840322 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840372 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840411 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840437 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znvq\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840465 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840507 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840567 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840597 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.840749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.840653 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.841144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.841038 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.841144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.841062 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.841144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.841106 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.841316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.841221 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.841510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.841484 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.843478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.843457 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.843661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.843640 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.901579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.901544 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znvq\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.901740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.901546 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.964760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.964717 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:16.965116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.965094 5275 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" secret="" err="secret \"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\" not found" Nov 03 04:25:16.965217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.965188 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:16.987552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:16.987516 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7d5936-3b42-4299-bef7-050cee937bfb" path="/var/lib/kubelet/pods/3c7d5936-3b42-4299-bef7-050cee937bfb/volumes" Nov 03 04:25:17.401974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.401942 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:17.403449 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:25:17.403418 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4e0c35_40ef_4144_a55d_06d10b64348e.slice/crio-27a7209fa55146cdfb4aeacefcd6ea367f0a01ddeec5081e80235a1c0dd86ec5 WatchSource:0}: Error finding container 27a7209fa55146cdfb4aeacefcd6ea367f0a01ddeec5081e80235a1c0dd86ec5: Status 404 returned error can't find the container with id 27a7209fa55146cdfb4aeacefcd6ea367f0a01ddeec5081e80235a1c0dd86ec5 Nov 03 04:25:17.405915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.405880 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:25:17.406005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.405949 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:25:17.406005 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.405983 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:25:17.597583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.597545 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" event={"ID":"8e4e0c35-40ef-4144-a55d-06d10b64348e","Type":"ContainerStarted","Data":"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de"} Nov 03 04:25:17.597781 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.597607 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" event={"ID":"8e4e0c35-40ef-4144-a55d-06d10b64348e","Type":"ContainerStarted","Data":"27a7209fa55146cdfb4aeacefcd6ea367f0a01ddeec5081e80235a1c0dd86ec5"} Nov 03 04:25:17.597781 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.597623 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" podUID="8e4e0c35-40ef-4144-a55d-06d10b64348e" containerName="istio-proxy" containerID="cri-o://b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de" gracePeriod=30 Nov 03 04:25:17.719653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:17.719543 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" podStartSLOduration=1.719527477 podStartE2EDuration="1.719527477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:25:17.717361086 +0000 UTC m=+5008.870049508" watchObservedRunningTime="2026-11-03 04:25:17.719527477 +0000 UTC m=+5008.872215898" Nov 03 04:25:18.805304 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:18.805258 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:25:22.865859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:22.865762 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:23.007714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007684 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.007904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007783 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.007904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.007904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007839 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.007904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007897 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.008122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007934 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.008122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.007978 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.008122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008029 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.008122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008072 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\") pod \"8e4e0c35-40ef-4144-a55d-06d10b64348e\" (UID: \"8e4e0c35-40ef-4144-a55d-06d10b64348e\") " Nov 03 04:25:23.008330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008227 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:23.008330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008268 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:23.008443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008424 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:23.008547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008528 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.008620 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008556 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4e0c35-40ef-4144-a55d-06d10b64348e-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.008620 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008570 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.008731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008644 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:23.008731 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.008678 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:23.010527 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.010507 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:25:23.010624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.010602 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:23.010725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.010708 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:23.010818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.010802 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq" pod "8e4e0c35-40ef-4144-a55d-06d10b64348e" (UID: "8e4e0c35-40ef-4144-a55d-06d10b64348e"). InnerVolumeSpecName "kube-api-access-8znvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:23.109645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109604 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.109645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109639 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8znvq\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-kube-api-access-8znvq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.109645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109650 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.109927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109659 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.109927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109667 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4e0c35-40ef-4144-a55d-06d10b64348e-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.109927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.109677 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4e0c35-40ef-4144-a55d-06d10b64348e-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:23.633591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.633556 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e4e0c35-40ef-4144-a55d-06d10b64348e" containerID="b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de" exitCode=0 Nov 03 04:25:23.633777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.633607 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" Nov 03 04:25:23.633777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.633638 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" event={"ID":"8e4e0c35-40ef-4144-a55d-06d10b64348e","Type":"ContainerDied","Data":"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de"} Nov 03 04:25:23.633777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.633697 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg" event={"ID":"8e4e0c35-40ef-4144-a55d-06d10b64348e","Type":"ContainerDied","Data":"27a7209fa55146cdfb4aeacefcd6ea367f0a01ddeec5081e80235a1c0dd86ec5"} Nov 03 04:25:23.633777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.633726 5275 scope.go:122] "RemoveContainer" containerID="b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de" Nov 03 04:25:23.645608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.645589 5275 scope.go:122] "RemoveContainer" containerID="b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de" Nov 03 04:25:23.645959 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:23.645937 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de\": container with ID starting with b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de not found: ID does not exist" containerID="b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de" Nov 03 04:25:23.646040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.645969 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de"} err="failed to get container status \"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de\": rpc error: code = NotFound desc = could not find container \"b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de\": container with ID starting with b3bb453a8406d41023ab14981f832c6eed7fa69fbe545f9b488c051b994550de not found: ID does not exist" Nov 03 04:25:23.845224 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.845189 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:23.895130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:23.895049 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-djkxg"] Nov 03 04:25:24.992023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:24.991985 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4e0c35-40ef-4144-a55d-06d10b64348e" path="/var/lib/kubelet/pods/8e4e0c35-40ef-4144-a55d-06d10b64348e/volumes" Nov 03 04:25:28.806048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:28.806010 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:25:31.175827 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.175797 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 04:25:31.194876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.194831 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\") pod \"36f8aacf-0968-44c1-a39e-db172cba4e3d\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " Nov 03 04:25:31.195034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.194999 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\" (UniqueName: \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\") pod \"36f8aacf-0968-44c1-a39e-db172cba4e3d\" (UID: \"36f8aacf-0968-44c1-a39e-db172cba4e3d\") " Nov 03 04:25:31.195226 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.195202 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume" pod "36f8aacf-0968-44c1-a39e-db172cba4e3d" (UID: "36f8aacf-0968-44c1-a39e-db172cba4e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:31.197266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.197235 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z" pod "36f8aacf-0968-44c1-a39e-db172cba4e3d" (UID: "36f8aacf-0968-44c1-a39e-db172cba4e3d"). InnerVolumeSpecName "kube-api-access-twj7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:31.296054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.296018 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twj7z\" (UniqueName: \"kubernetes.io/projected/36f8aacf-0968-44c1-a39e-db172cba4e3d-kube-api-access-twj7z\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:31.296054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.296054 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f8aacf-0968-44c1-a39e-db172cba4e3d-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:31.683524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.683490 5275 generic.go:356] "Generic (PLEG): container finished" podID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerID="9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a" exitCode=0 Nov 03 04:25:31.683753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.683548 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" event={"ID":"36f8aacf-0968-44c1-a39e-db172cba4e3d","Type":"ContainerDied","Data":"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a"} Nov 03 04:25:31.683753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.683591 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" event={"ID":"36f8aacf-0968-44c1-a39e-db172cba4e3d","Type":"ContainerDied","Data":"ee4b6807b37665d0619180f1d277e2b2fe53ea4b5302044ed76c9853cbbb8df3"} Nov 03 04:25:31.683753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.683616 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh" Nov 03 04:25:31.683753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.683629 5275 scope.go:122] "RemoveContainer" containerID="9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a" Nov 03 04:25:31.694381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.694357 5275 scope.go:122] "RemoveContainer" containerID="9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a" Nov 03 04:25:31.694626 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:31.694605 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a\": container with ID starting with 9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a not found: ID does not exist" containerID="9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a" Nov 03 04:25:31.694703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.694638 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a"} err="failed to get container status \"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a\": rpc error: code = NotFound desc = could not find container \"9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a\": container with ID starting with 9ff9a1bc7ced79dae0b1cc4cd3e34b813ed93401a2c09a7e5403dce7948c4e9a not found: ID does not exist" Nov 03 04:25:31.849167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.849133 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 04:25:31.853328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:31.853298 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-66d902-5c9cbfd6ff-zgqqh"] Nov 03 04:25:32.335710 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:32.335669 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:25:32.985143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:32.985103 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" path="/var/lib/kubelet/pods/36f8aacf-0968-44c1-a39e-db172cba4e3d/volumes" Nov 03 04:25:38.655263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:38.655216 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 04:25:38.655800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:38.655527 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" containerID="cri-o://3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d" gracePeriod=30 Nov 03 04:25:39.841594 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.841555 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4"] Nov 03 04:25:39.842058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842000 5275 cpu_manager.go:414] "containerMap: removing container" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" Nov 03 04:25:39.842058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842012 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" Nov 03 04:25:39.842058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842031 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e4e0c35-40ef-4144-a55d-06d10b64348e" containerName="istio-proxy" Nov 03 04:25:39.842058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842036 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4e0c35-40ef-4144-a55d-06d10b64348e" containerName="istio-proxy" Nov 03 04:25:39.842326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842114 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f8aacf-0968-44c1-a39e-db172cba4e3d" containerName="payload-processing" Nov 03 04:25:39.842326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.842129 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4e0c35-40ef-4144-a55d-06d10b64348e" containerName="istio-proxy" Nov 03 04:25:39.846810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.846792 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.889087 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.889044 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.889283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.889127 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bvqn\" (UniqueName: \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.889283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.889220 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.900066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.900012 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:25:39.903337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.903306 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4"] Nov 03 04:25:39.949380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.949336 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:25:39.949380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.949346 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:25:39.949611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.949403 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:25:39.990753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.990715 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.990974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.990790 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bvqn\" (UniqueName: \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.990974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.990825 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.991466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.991436 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:39.993439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:39.993419 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:40.052223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:40.052192 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bvqn\" (UniqueName: \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\") pod \"maas-api-revoke-sub-927a8430-6ngx4\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:40.169913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:40.169781 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:40.556505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:40.556475 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4"] Nov 03 04:25:40.558906 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:25:40.558837 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1a31d7_6ee9_4a16_98c0_14534585459b.slice/crio-16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c WatchSource:0}: Error finding container 16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c: Status 404 returned error can't find the container with id 16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c Nov 03 04:25:40.741343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:40.741302 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" event={"ID":"8e1a31d7-6ee9-4a16-98c0-14534585459b","Type":"ContainerStarted","Data":"16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c"} Nov 03 04:25:41.749695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:41.749593 5275 generic.go:356] "Generic (PLEG): container finished" podID="8e1a31d7-6ee9-4a16-98c0-14534585459b" containerID="46699fee036493ce3a4275c42869457b227ff1b6d92bee3b80eb4c34a3775811" exitCode=0 Nov 03 04:25:41.750096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:41.749685 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" event={"ID":"8e1a31d7-6ee9-4a16-98c0-14534585459b","Type":"ContainerDied","Data":"46699fee036493ce3a4275c42869457b227ff1b6d92bee3b80eb4c34a3775811"} Nov 03 04:25:42.911492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:42.911466 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:43.023112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.023080 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\" (UniqueName: \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\") pod \"8e1a31d7-6ee9-4a16-98c0-14534585459b\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " Nov 03 04:25:43.023298 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.023132 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\") pod \"8e1a31d7-6ee9-4a16-98c0-14534585459b\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " Nov 03 04:25:43.023298 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.023176 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\") pod \"8e1a31d7-6ee9-4a16-98c0-14534585459b\" (UID: \"8e1a31d7-6ee9-4a16-98c0-14534585459b\") " Nov 03 04:25:43.023546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.023519 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca" pod "8e1a31d7-6ee9-4a16-98c0-14534585459b" (UID: "8e1a31d7-6ee9-4a16-98c0-14534585459b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:43.025525 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.025501 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn" pod "8e1a31d7-6ee9-4a16-98c0-14534585459b" (UID: "8e1a31d7-6ee9-4a16-98c0-14534585459b"). InnerVolumeSpecName "kube-api-access-9bvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:43.025632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.025559 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca" pod "8e1a31d7-6ee9-4a16-98c0-14534585459b" (UID: "8e1a31d7-6ee9-4a16-98c0-14534585459b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:43.124078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.124042 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bvqn\" (UniqueName: \"kubernetes.io/projected/8e1a31d7-6ee9-4a16-98c0-14534585459b-kube-api-access-9bvqn\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:43.124078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.124072 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:43.124078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.124083 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1a31d7-6ee9-4a16-98c0-14534585459b-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:43.768795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.768673 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" Nov 03 04:25:43.768795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.768705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4" event={"ID":"8e1a31d7-6ee9-4a16-98c0-14534585459b","Type":"ContainerDied","Data":"16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c"} Nov 03 04:25:43.768795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.768768 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16fc16f5f0dfd09b2110f8074e2e04ea4bb41736a42b303ffff84d05141bf03c" Nov 03 04:25:43.930541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:43.930465 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 04:25:44.031609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031578 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.031793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031641 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.031793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031684 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.031793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031711 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.031793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031785 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.032074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031815 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.032074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031868 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.032074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.031953 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.032074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032003 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\") pod \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\" (UID: \"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56\") " Nov 03 04:25:44.032311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032137 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:44.032311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032216 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:44.032311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032257 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:44.032587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032563 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:44.032684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032570 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.032684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032618 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.032684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.032639 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.033227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.033199 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:44.034355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.034333 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:25:44.034860 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.034818 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "kube-api-access-gcpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:44.035045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.035025 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:44.035336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.035320 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy" pod "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" (UID: "ca18c3b7-de5d-44c3-adcb-b6ad1df26a56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:44.133240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133202 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.133240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133235 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcpts\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-kube-api-access-gcpts\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.133240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133245 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.133506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133254 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.133506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133264 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.133506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.133272 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:44.152552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.152518 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 04:25:44.163170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.163135 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 04:25:44.775644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.775604 5275 generic.go:356] "Generic (PLEG): container finished" podID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerID="3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d" exitCode=0 Nov 03 04:25:44.775909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.775649 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerDied","Data":"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d"} Nov 03 04:25:44.775909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.775700 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" Nov 03 04:25:44.775909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.775755 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq" event={"ID":"ca18c3b7-de5d-44c3-adcb-b6ad1df26a56","Type":"ContainerDied","Data":"16ffd7283fefe4812bd165f16736533bf0ce8c532806d3645a183618a34df6f7"} Nov 03 04:25:44.775909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.775789 5275 scope.go:122] "RemoveContainer" containerID="3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d" Nov 03 04:25:44.786809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.786423 5275 scope.go:122] "RemoveContainer" containerID="7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029" Nov 03 04:25:44.795822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.795803 5275 scope.go:122] "RemoveContainer" containerID="3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d" Nov 03 04:25:44.796130 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:44.796107 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d\": container with ID starting with 3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d not found: ID does not exist" containerID="3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d" Nov 03 04:25:44.796179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.796141 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d"} err="failed to get container status \"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d\": rpc error: code = NotFound desc = could not find container \"3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d\": container with ID starting with 3dbaaee2eaba60f34d599cdd36aa5e2259efe64a8028d5780ef195c8f278a07d not found: ID does not exist" Nov 03 04:25:44.796179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.796160 5275 scope.go:122] "RemoveContainer" containerID="7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029" Nov 03 04:25:44.796431 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:44.796412 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029\": container with ID starting with 7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029 not found: ID does not exist" containerID="7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029" Nov 03 04:25:44.796493 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.796437 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029"} err="failed to get container status \"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029\": rpc error: code = NotFound desc = could not find container \"7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029\": container with ID starting with 7abe15b844e5d81c94324d5294f2dc0f7cb81a4e9c6fdda0f45abf913cc25029 not found: ID does not exist" Nov 03 04:25:44.941081 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.941037 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 04:25:44.991497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:44.991461 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3d8dc5-openshift-default-cf99d9f5-s4jjq"] Nov 03 04:25:46.983460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:46.983427 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" path="/var/lib/kubelet/pods/ca18c3b7-de5d-44c3-adcb-b6ad1df26a56/volumes" Nov 03 04:25:57.032854 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.032805 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg"] Nov 03 04:25:57.033301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033263 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.033301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033277 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.033301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033285 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8e1a31d7-6ee9-4a16-98c0-14534585459b" containerName="revoke-keys" Nov 03 04:25:57.033301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033291 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1a31d7-6ee9-4a16-98c0-14534585459b" containerName="revoke-keys" Nov 03 04:25:57.033301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033302 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.033482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033308 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.033482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033371 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.033482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033377 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1a31d7-6ee9-4a16-98c0-14534585459b" containerName="revoke-keys" Nov 03 04:25:57.033482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.033384 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca18c3b7-de5d-44c3-adcb-b6ad1df26a56" containerName="istio-proxy" Nov 03 04:25:57.036595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.036576 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.139263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.139230 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:25:57.139263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.139231 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\"" Nov 03 04:25:57.139509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.139284 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:25:57.139509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.139308 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:25:57.142070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.142050 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg"] Nov 03 04:25:57.167283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.167246 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t57d\" (UniqueName: \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.167478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.167362 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.167478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.167393 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.268254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.268221 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t57d\" (UniqueName: \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.268487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.268314 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.268487 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.268354 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.269075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.269048 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.270994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.270973 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.329566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.329492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t57d\" (UniqueName: \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\") pod \"maas-api-revoke-sub-e9c9f3c9-x5nqg\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.349370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.349338 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:25:57.725297 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.725268 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg"] Nov 03 04:25:57.727032 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:25:57.727002 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9443e_2f73_41f4_9236_c409ff21b1e5.slice/crio-1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0 WatchSource:0}: Error finding container 1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0: Status 404 returned error can't find the container with id 1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0 Nov 03 04:25:57.854554 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:57.854520 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" event={"ID":"3aa9443e-2f73-41f4-9236-c409ff21b1e5","Type":"ContainerStarted","Data":"1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0"} Nov 03 04:25:58.862018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:58.861927 5275 generic.go:356] "Generic (PLEG): container finished" podID="3aa9443e-2f73-41f4-9236-c409ff21b1e5" containerID="de46f4d59c1304048b269176ad4556a74bf242a7a786e00fdb728a52f1a4f0ec" exitCode=0 Nov 03 04:25:58.862406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:58.862012 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" event={"ID":"3aa9443e-2f73-41f4-9236-c409ff21b1e5","Type":"ContainerDied","Data":"de46f4d59c1304048b269176ad4556a74bf242a7a786e00fdb728a52f1a4f0ec"} Nov 03 04:25:59.029010 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.028978 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 04:25:59.175115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.175017 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" containerID="cri-o://54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa" gracePeriod=45 Nov 03 04:25:59.184030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.183991 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" containerID="cri-o://25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3" gracePeriod=45 Nov 03 04:25:59.247618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.247581 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused" Nov 03 04:25:59.496111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.496081 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 04:25:59.514023 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.514000 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 04:25:59.593290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593254 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.593480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593322 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\" (UniqueName: \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.593480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593362 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\" (UniqueName: \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.593480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593428 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\" (UniqueName: \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.593480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593458 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593505 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593571 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593600 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593659 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593692 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.593761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593733 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.594127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593764 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\") pod \"803d934a-5742-443b-886a-7420addbd746\" (UID: \"803d934a-5742-443b-886a-7420addbd746\") " Nov 03 04:25:59.594127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.593789 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\") pod \"7b18951d-2a00-4f58-a3d0-5451c61ec179\" (UID: \"7b18951d-2a00-4f58-a3d0-5451c61ec179\") " Nov 03 04:25:59.594341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.594198 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.594874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.594699 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.594874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.594718 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.595061 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.594916 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.595127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.595084 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.595287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.595245 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.596908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.596833 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "kube-api-access-ptwfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:59.598068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.598018 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:59.598068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.598060 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm" pod "803d934a-5742-443b-886a-7420addbd746" (UID: "803d934a-5742-443b-886a-7420addbd746"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.598200 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.598068 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.599179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.599148 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "kube-api-access-fdj4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:59.600074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.600049 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs" pod "7b18951d-2a00-4f58-a3d0-5451c61ec179" (UID: "7b18951d-2a00-4f58-a3d0-5451c61ec179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:59.695516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695475 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695509 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695519 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695527 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803d934a-5742-443b-886a-7420addbd746-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695543 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdj4p\" (UniqueName: \"kubernetes.io/projected/7b18951d-2a00-4f58-a3d0-5451c61ec179-kube-api-access-fdj4p\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695552 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwfd\" (UniqueName: \"kubernetes.io/projected/803d934a-5742-443b-886a-7420addbd746-kube-api-access-ptwfd\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695560 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803d934a-5742-443b-886a-7420addbd746-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695572 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695579 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695588 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b18951d-2a00-4f58-a3d0-5451c61ec179-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.695779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.695596 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b18951d-2a00-4f58-a3d0-5451c61ec179-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.869146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.869108 5275 generic.go:356] "Generic (PLEG): container finished" podID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerID="54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa" exitCode=0 Nov 03 04:25:59.869603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.869152 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" Nov 03 04:25:59.869603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.869195 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" event={"ID":"7b18951d-2a00-4f58-a3d0-5451c61ec179","Type":"ContainerDied","Data":"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa"} Nov 03 04:25:59.869603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.869241 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" event={"ID":"7b18951d-2a00-4f58-a3d0-5451c61ec179","Type":"ContainerDied","Data":"26cb31bff94e1042c9090f1078fe2011d45f592c60bcaad6092bc498a1d1fccb"} Nov 03 04:25:59.869603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.869263 5275 scope.go:122] "RemoveContainer" containerID="54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa" Nov 03 04:25:59.870749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.870725 5275 generic.go:356] "Generic (PLEG): container finished" podID="803d934a-5742-443b-886a-7420addbd746" containerID="25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3" exitCode=0 Nov 03 04:25:59.870945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.870775 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" Nov 03 04:25:59.870945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.870808 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" event={"ID":"803d934a-5742-443b-886a-7420addbd746","Type":"ContainerDied","Data":"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3"} Nov 03 04:25:59.870945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.870881 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn" event={"ID":"803d934a-5742-443b-886a-7420addbd746","Type":"ContainerDied","Data":"40f05007e2bec4f63716c061f171c1b62978d7aa3914469cbd119967398935a9"} Nov 03 04:25:59.880463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.880440 5275 scope.go:122] "RemoveContainer" containerID="54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa" Nov 03 04:25:59.880810 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:59.880789 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa\": container with ID starting with 54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa not found: ID does not exist" containerID="54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa" Nov 03 04:25:59.880931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.880822 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa"} err="failed to get container status \"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa\": rpc error: code = NotFound desc = could not find container \"54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa\": container with ID starting with 54f0e5672652ffeee038f39f5819e6227138aed76b323916cef15d9aea8c28fa not found: ID does not exist" Nov 03 04:25:59.880931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.880863 5275 scope.go:122] "RemoveContainer" containerID="25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3" Nov 03 04:25:59.891399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.891379 5275 scope.go:122] "RemoveContainer" containerID="25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3" Nov 03 04:25:59.891705 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:25:59.891683 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3\": container with ID starting with 25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3 not found: ID does not exist" containerID="25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3" Nov 03 04:25:59.891821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.891711 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3"} err="failed to get container status \"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3\": rpc error: code = NotFound desc = could not find container \"25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3\": container with ID starting with 25fbbb03e2d162d70dbd70c287b5ef0338337dba350d678bf1ada613588f37c3 not found: ID does not exist" Nov 03 04:25:59.998810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:25:59.998782 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:26:00.033948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.033911 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 04:26:00.038966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.038927 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw"] Nov 03 04:26:00.099323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.099289 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\") pod \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " Nov 03 04:26:00.099537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.099351 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\" (UniqueName: \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\") pod \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " Nov 03 04:26:00.099537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.099373 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\") pod \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\" (UID: \"3aa9443e-2f73-41f4-9236-c409ff21b1e5\") " Nov 03 04:26:00.099728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.099698 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca" pod "3aa9443e-2f73-41f4-9236-c409ff21b1e5" (UID: "3aa9443e-2f73-41f4-9236-c409ff21b1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:26:00.101707 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.101682 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca" pod "3aa9443e-2f73-41f4-9236-c409ff21b1e5" (UID: "3aa9443e-2f73-41f4-9236-c409ff21b1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:26:00.101772 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.101688 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d" pod "3aa9443e-2f73-41f4-9236-c409ff21b1e5" (UID: "3aa9443e-2f73-41f4-9236-c409ff21b1e5"). InnerVolumeSpecName "kube-api-access-4t57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:26:00.149050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.148961 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 04:26:00.152707 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.152679 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/simulator-w2-de27f5-kserve-56bd87cc7d-gptrn"] Nov 03 04:26:00.201270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.201240 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:00.201270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.201268 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t57d\" (UniqueName: \"kubernetes.io/projected/3aa9443e-2f73-41f4-9236-c409ff21b1e5-kube-api-access-4t57d\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:00.201486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.201281 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9443e-2f73-41f4-9236-c409ff21b1e5-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:00.419478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.419375 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/premium-w2-de27f5-kserve-6dc56cbf47-s7hrw" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 04:26:00.878207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.878172 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" Nov 03 04:26:00.878207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.878195 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg" event={"ID":"3aa9443e-2f73-41f4-9236-c409ff21b1e5","Type":"ContainerDied","Data":"1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0"} Nov 03 04:26:00.878782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.878269 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de89ce3618ec45c82bdb491cdac5d98da3b5291fe75cc946069bf11179ad7e0" Nov 03 04:26:00.983488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.983452 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" path="/var/lib/kubelet/pods/7b18951d-2a00-4f58-a3d0-5451c61ec179/volumes" Nov 03 04:26:00.983928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:00.983913 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803d934a-5742-443b-886a-7420addbd746" path="/var/lib/kubelet/pods/803d934a-5742-443b-886a-7420addbd746/volumes" Nov 03 04:26:02.330117 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:26:02.330083 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:26:04.029420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:04.029381 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" containerID="cri-o://e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1" gracePeriod=25 Nov 03 04:26:08.506341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:08.506299 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:26:16.052917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:16.052881 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v"] Nov 03 04:26:16.101682 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:16.101643 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076d139b-gps5v"] Nov 03 04:26:16.982780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:16.982734 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76aa0d8f-2157-4f7c-8f5c-058bf112bf3d" path="/var/lib/kubelet/pods/76aa0d8f-2157-4f7c-8f5c-058bf112bf3d/volumes" Nov 03 04:26:17.145742 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.145706 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:17.146189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146171 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3aa9443e-2f73-41f4-9236-c409ff21b1e5" containerName="revoke-keys" Nov 03 04:26:17.146248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146192 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa9443e-2f73-41f4-9236-c409ff21b1e5" containerName="revoke-keys" Nov 03 04:26:17.146248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146213 5275 cpu_manager.go:414] "containerMap: removing container" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" Nov 03 04:26:17.146248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146222 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" Nov 03 04:26:17.146374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146261 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" Nov 03 04:26:17.146374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146270 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" Nov 03 04:26:17.146468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146417 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa9443e-2f73-41f4-9236-c409ff21b1e5" containerName="revoke-keys" Nov 03 04:26:17.146468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146432 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="803d934a-5742-443b-886a-7420addbd746" containerName="main" Nov 03 04:26:17.146468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.146450 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b18951d-2a00-4f58-a3d0-5451c61ec179" containerName="main" Nov 03 04:26:17.153930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.152643 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.206400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.206367 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\"" Nov 03 04:26:17.259711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.259629 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:17.279698 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.279654 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.279912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.279706 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.279912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.279743 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.279930 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.279986 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7n4r\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.280108 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.280187 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.280300 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.280584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.280346 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.381975 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382062 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382103 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382145 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382177 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382206 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382472 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382711 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.382952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382924 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.383034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.382980 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.383034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.383022 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n4r\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.383152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.383081 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.383442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.383418 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.383507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.383462 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.384821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.384798 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.384954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.384913 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.443586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.443551 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.443826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.443806 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n4r\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.472138 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.472100 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:17.905997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.905963 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:17.907948 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:26:17.907917 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0da82ff_814f_4d92_b9ba_c90337716dc2.slice/crio-62ad518536b83e75e56c9d0730017fa1c84166dc2d22f0f1630951affc2e14df WatchSource:0}: Error finding container 62ad518536b83e75e56c9d0730017fa1c84166dc2d22f0f1630951affc2e14df: Status 404 returned error can't find the container with id 62ad518536b83e75e56c9d0730017fa1c84166dc2d22f0f1630951affc2e14df Nov 03 04:26:17.910244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.910209 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:26:17.910343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.910278 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:26:17.910343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.910307 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:26:17.980594 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.980552 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" event={"ID":"c0da82ff-814f-4d92-b9ba-c90337716dc2","Type":"ContainerStarted","Data":"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba"} Nov 03 04:26:17.980822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:17.980801 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" event={"ID":"c0da82ff-814f-4d92-b9ba-c90337716dc2","Type":"ContainerStarted","Data":"62ad518536b83e75e56c9d0730017fa1c84166dc2d22f0f1630951affc2e14df"} Nov 03 04:26:18.103752 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:18.103692 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podStartSLOduration=1.103679079 podStartE2EDuration="1.103679079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:26:18.100411306 +0000 UTC m=+5069.253099726" watchObservedRunningTime="2026-11-03 04:26:18.103679079 +0000 UTC m=+5069.256367499" Nov 03 04:26:18.474354 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:18.473920 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:18.474913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:18.474816 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:18.474969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:18.474901 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:18.505965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:18.505926 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:26:19.474386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:19.474339 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:19.474903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:19.474408 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:20.474349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:20.474302 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:20.474567 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:20.474388 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:21.478065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:21.478023 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:21.478765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:21.478726 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:22.473320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:22.473278 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:22.473643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:22.473612 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:23.474318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:23.474270 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:23.474896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:23.474346 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:24.307941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.307911 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 04:26:24.372679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.372641 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\" (UniqueName: \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\") pod \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " Nov 03 04:26:24.372924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.372775 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\") pod \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\" (UID: \"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0\") " Nov 03 04:26:24.373218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.373194 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume" pod "fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" (UID: "fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:26:24.374942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.374912 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst" pod "fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" (UID: "fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0"). InnerVolumeSpecName "kube-api-access-kgnst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:26:24.473316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.473272 5275 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body= Nov 03 04:26:24.473523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.473343 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" Nov 03 04:26:24.473787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.473761 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgnst\" (UniqueName: \"kubernetes.io/projected/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-kube-api-access-kgnst\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:24.473897 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:24.473793 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:25.010355 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:26:25.010309 5275 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc744aa1_fa60_4f2a_9b1e_e625bfd7a9e0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc744aa1_fa60_4f2a_9b1e_e625bfd7a9e0.slice/crio-145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503\": RecentStats: unable to find data in memory cache]" Nov 03 04:26:25.022510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.022478 5275 generic.go:356] "Generic (PLEG): container finished" podID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerID="e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1" exitCode=0 Nov 03 04:26:25.022669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.022544 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" Nov 03 04:26:25.022669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.022571 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" event={"ID":"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0","Type":"ContainerDied","Data":"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1"} Nov 03 04:26:25.022669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.022647 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss" event={"ID":"fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0","Type":"ContainerDied","Data":"145e6c54919726a5528465e575bfcaeae52d5475b5b382e106324cb43c6ec503"} Nov 03 04:26:25.022820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.022684 5275 scope.go:122] "RemoveContainer" containerID="e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1" Nov 03 04:26:25.034456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.034429 5275 scope.go:122] "RemoveContainer" containerID="e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1" Nov 03 04:26:25.034822 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:26:25.034802 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1\": container with ID starting with e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1 not found: ID does not exist" containerID="e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1" Nov 03 04:26:25.034912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.034831 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1"} err="failed to get container status \"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1\": rpc error: code = NotFound desc = could not find container \"e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1\": container with ID starting with e3b56667a53be2b470f7372f170620baf423f21aaf57eff9fb29ad7b9a67d3f1 not found: ID does not exist" Nov 03 04:26:25.191208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.191160 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 04:26:25.193413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.193388 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de27f5-68cb978b75-f5jss"] Nov 03 04:26:25.701929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.701892 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:25.702358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.702341 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:25.703879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:25.703838 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:26.825703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:26.825668 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:26.982967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:26.982937 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" path="/var/lib/kubelet/pods/fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0/volumes" Nov 03 04:26:27.035785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:27.035751 5275 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" secret="" err="secret \"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\" not found" Nov 03 04:26:27.047447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:27.047415 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p"] Nov 03 04:26:27.052265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:27.052235 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0144e8b-txq5p"] Nov 03 04:26:28.041497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:28.041457 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" containerID="cri-o://dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba" gracePeriod=30 Nov 03 04:26:28.081374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:28.081338 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c66d77cd-926q9_178bdf09-2334-43af-93f3-175fc348c04a/authorino/0.log" Nov 03 04:26:28.984379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:28.984338 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc30d173-29dd-4c73-95b5-78046c770fa5" path="/var/lib/kubelet/pods/cc30d173-29dd-4c73-95b5-78046c770fa5/volumes" Nov 03 04:26:29.541245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:29.541210 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nvhkn_11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04/manager/0.log" Nov 03 04:26:30.120086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:30.120056 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z9kkw_a60f9acc-e4f3-46e1-9951-f0fb96818e0d/manager/0.log" Nov 03 04:26:31.766725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:31.766690 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l6m8z_c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec/manager/0.log" Nov 03 04:26:32.331185 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:26:32.331150 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:26:33.293460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.293433 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:33.371352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371253 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371315 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371361 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371407 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371472 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371508 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371536 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371611 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371679 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\") pod \"c0da82ff-814f-4d92-b9ba-c90337716dc2\" (UID: \"c0da82ff-814f-4d92-b9ba-c90337716dc2\") " Nov 03 04:26:33.371924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371875 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:26:33.371924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.371892 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:26:33.372318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.372041 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:26:33.372318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.372094 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.372318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.372106 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:26:33.372318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.372115 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.372565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.372476 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:26:33.373830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.373803 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:26:33.373830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.373815 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:26:33.374239 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.374223 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:26:33.374539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.374524 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r" pod "c0da82ff-814f-4d92-b9ba-c90337716dc2" (UID: "c0da82ff-814f-4d92-b9ba-c90337716dc2"). InnerVolumeSpecName "kube-api-access-v7n4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:26:33.472759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472727 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.472759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472755 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.472759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472765 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.473025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472773 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.473025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472782 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da82ff-814f-4d92-b9ba-c90337716dc2-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.473025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472790 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7n4r\" (UniqueName: \"kubernetes.io/projected/c0da82ff-814f-4d92-b9ba-c90337716dc2-kube-api-access-v7n4r\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.473025 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.472799 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da82ff-814f-4d92-b9ba-c90337716dc2-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:33.842797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:33.842765 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lskgg_f0d5b72d-3781-4776-8dfe-4b2728d5d42e/server/0.log" Nov 03 04:26:34.075098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.075059 5275 generic.go:356] "Generic (PLEG): container finished" podID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerID="dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba" exitCode=0 Nov 03 04:26:34.075293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.075098 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" event={"ID":"c0da82ff-814f-4d92-b9ba-c90337716dc2","Type":"ContainerDied","Data":"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba"} Nov 03 04:26:34.075293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.075114 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" Nov 03 04:26:34.075293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.075161 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59" event={"ID":"c0da82ff-814f-4d92-b9ba-c90337716dc2","Type":"ContainerDied","Data":"62ad518536b83e75e56c9d0730017fa1c84166dc2d22f0f1630951affc2e14df"} Nov 03 04:26:34.075293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.075192 5275 scope.go:122] "RemoveContainer" containerID="dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba" Nov 03 04:26:34.085251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.085221 5275 scope.go:122] "RemoveContainer" containerID="dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba" Nov 03 04:26:34.085538 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:26:34.085516 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba\": container with ID starting with dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba not found: ID does not exist" containerID="dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba" Nov 03 04:26:34.085617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.085553 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba"} err="failed to get container status \"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba\": rpc error: code = NotFound desc = could not find container \"dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba\": container with ID starting with dbf933fd52bfdcff49fb6bb3eede74c8e09c5c8e3902d419a7eddba4a8c92aba not found: ID does not exist" Nov 03 04:26:34.293416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.293379 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:34.297788 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.297760 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-j4j59"] Nov 03 04:26:34.984015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:34.983980 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" path="/var/lib/kubelet/pods/c0da82ff-814f-4d92-b9ba-c90337716dc2/volumes" Nov 03 04:26:46.675055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675016 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:26:46.675480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675456 5275 cpu_manager.go:414] "containerMap: removing container" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" Nov 03 04:26:46.675480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675470 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" Nov 03 04:26:46.675565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675496 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" Nov 03 04:26:46.675565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675501 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" Nov 03 04:26:46.675639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675566 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc744aa1-fa60-4f2a-9b1e-e625bfd7a9e0" containerName="payload-pre-processing" Nov 03 04:26:46.675639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.675580 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0da82ff-814f-4d92-b9ba-c90337716dc2" containerName="istio-proxy" Nov 03 04:26:46.680242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.680225 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.736920 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.736879 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:26:46.782043 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.782014 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-246eeedd\"" Nov 03 04:26:46.804690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.804649 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9cpb\" (UniqueName: \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.804903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.804833 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.905780 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.905740 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.905993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.905875 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9cpb\" (UniqueName: \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.907094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.907068 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.966884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.966791 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9cpb\" (UniqueName: \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\") pod \"payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:46.994624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:46.994584 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:47.370672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:47.370641 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:26:47.372487 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:26:47.372458 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4815ea7_da30_4139_affe_55db650564e8.slice/crio-843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18 WatchSource:0}: Error finding container 843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18: Status 404 returned error can't find the container with id 843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18 Nov 03 04:26:48.158065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:48.158031 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" event={"ID":"b4815ea7-da30-4139-affe-55db650564e8","Type":"ContainerStarted","Data":"3f2a4be9ac2647b25a129dbb2be925ded9f0604b63a807447ef7f72e9a00b5f0"} Nov 03 04:26:48.158578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:48.158090 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" event={"ID":"b4815ea7-da30-4139-affe-55db650564e8","Type":"ContainerStarted","Data":"843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18"} Nov 03 04:26:48.158578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:48.158178 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:26:48.276075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:48.276017 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" podStartSLOduration=2.276002513 podStartE2EDuration="2.276002513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:26:48.273133803 +0000 UTC m=+5099.425822224" watchObservedRunningTime="2026-11-03 04:26:48.276002513 +0000 UTC m=+5099.428690934" Nov 03 04:26:50.490506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:50.490473 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:26:50.528315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:50.528288 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:26:59.168773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:26:59.168743 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:27:02.328591 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:27:02.328551 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:27:19.603900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.603864 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r"] Nov 03 04:27:19.607668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.607651 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.659774 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.659730 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:27:19.709226 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.709194 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:27:19.709226 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.709201 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:27:19.709486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.709425 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:27:19.711702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.711680 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r"] Nov 03 04:27:19.730545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.730500 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.730724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.730572 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.730724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.730665 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzlk\" (UniqueName: \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.831806 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.831765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.832024 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.831834 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.832024 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.831885 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzlk\" (UniqueName: \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.832577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.832549 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.834317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.834297 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.893491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.893421 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzlk\" (UniqueName: \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\") pod \"maas-api-revoke-keys-e2e-ait-246eeedd-f222r\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:19.921453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:19.921411 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:20.300328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:20.300301 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r"] Nov 03 04:27:20.302825 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:27:20.302794 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef629c93_86ad_4508_9eac_653154172bfa.slice/crio-d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3 WatchSource:0}: Error finding container d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3: Status 404 returned error can't find the container with id d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3 Nov 03 04:27:20.337034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:20.337002 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" event={"ID":"ef629c93-86ad-4508-9eac-653154172bfa","Type":"ContainerStarted","Data":"d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3"} Nov 03 04:27:21.348660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:21.348625 5275 generic.go:356] "Generic (PLEG): container finished" podID="ef629c93-86ad-4508-9eac-653154172bfa" containerID="7904215625f846b9f51a0682d3acc366c82384d5f6f9d04387416a722cd2a627" exitCode=0 Nov 03 04:27:21.349120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:21.348666 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" event={"ID":"ef629c93-86ad-4508-9eac-653154172bfa","Type":"ContainerDied","Data":"7904215625f846b9f51a0682d3acc366c82384d5f6f9d04387416a722cd2a627"} Nov 03 04:27:22.489326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.489293 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:22.557111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.557071 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\") pod \"ef629c93-86ad-4508-9eac-653154172bfa\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " Nov 03 04:27:22.557320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.557175 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\" (UniqueName: \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\") pod \"ef629c93-86ad-4508-9eac-653154172bfa\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " Nov 03 04:27:22.557320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.557250 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\") pod \"ef629c93-86ad-4508-9eac-653154172bfa\" (UID: \"ef629c93-86ad-4508-9eac-653154172bfa\") " Nov 03 04:27:22.557614 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.557581 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca" pod "ef629c93-86ad-4508-9eac-653154172bfa" (UID: "ef629c93-86ad-4508-9eac-653154172bfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:27:22.559518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.559491 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca" pod "ef629c93-86ad-4508-9eac-653154172bfa" (UID: "ef629c93-86ad-4508-9eac-653154172bfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:27:22.559625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.559545 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk" pod "ef629c93-86ad-4508-9eac-653154172bfa" (UID: "ef629c93-86ad-4508-9eac-653154172bfa"). InnerVolumeSpecName "kube-api-access-bpzlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:27:22.658223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.658140 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:22.658223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.658171 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef629c93-86ad-4508-9eac-653154172bfa-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:22.658223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:22.658181 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzlk\" (UniqueName: \"kubernetes.io/projected/ef629c93-86ad-4508-9eac-653154172bfa-kube-api-access-bpzlk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:23.360706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:23.360673 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" Nov 03 04:27:23.360918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:23.360710 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r" event={"ID":"ef629c93-86ad-4508-9eac-653154172bfa","Type":"ContainerDied","Data":"d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3"} Nov 03 04:27:23.360918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:23.360758 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cea741ff16b76b18e2f468773b56804e5384cfbf7d0f99a0faf959e38540c3" Nov 03 04:27:31.197059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:31.197020 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:27:32.330473 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:27:32.330434 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:27:33.050265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:33.050230 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d"] Nov 03 04:27:33.099838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:33.099802 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fe86ea-2xp7d"] Nov 03 04:27:34.983445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:34.983411 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1052bd4-cd1c-47f4-aea2-11969daf75f6" path="/var/lib/kubelet/pods/a1052bd4-cd1c-47f4-aea2-11969daf75f6/volumes" Nov 03 04:27:35.054340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:35.054303 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv"] Nov 03 04:27:35.103266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:35.103232 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8b49f89-6fshv"] Nov 03 04:27:36.197936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.197885 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" podUID="b4815ea7-da30-4139-affe-55db650564e8" containerName="payload-pre-processing" containerID="cri-o://3f2a4be9ac2647b25a129dbb2be925ded9f0604b63a807447ef7f72e9a00b5f0" gracePeriod=25 Nov 03 04:27:36.440895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.440865 5275 generic.go:356] "Generic (PLEG): container finished" podID="b4815ea7-da30-4139-affe-55db650564e8" containerID="3f2a4be9ac2647b25a129dbb2be925ded9f0604b63a807447ef7f72e9a00b5f0" exitCode=0 Nov 03 04:27:36.441029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.440888 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" event={"ID":"b4815ea7-da30-4139-affe-55db650564e8","Type":"ContainerDied","Data":"3f2a4be9ac2647b25a129dbb2be925ded9f0604b63a807447ef7f72e9a00b5f0"} Nov 03 04:27:36.441029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.440953 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" event={"ID":"b4815ea7-da30-4139-affe-55db650564e8","Type":"ContainerDied","Data":"843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18"} Nov 03 04:27:36.441029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.440978 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843eed4168d2319fc57de9d2a0866c8218ee87178da8f39603fcc1cbd9e5ba18" Nov 03 04:27:36.447302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.447280 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:27:36.596128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.596094 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\" (UniqueName: \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\") pod \"b4815ea7-da30-4139-affe-55db650564e8\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " Nov 03 04:27:36.596318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.596259 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\") pod \"b4815ea7-da30-4139-affe-55db650564e8\" (UID: \"b4815ea7-da30-4139-affe-55db650564e8\") " Nov 03 04:27:36.596640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.596616 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume" pod "b4815ea7-da30-4139-affe-55db650564e8" (UID: "b4815ea7-da30-4139-affe-55db650564e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:27:36.598337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.598308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb" pod "b4815ea7-da30-4139-affe-55db650564e8" (UID: "b4815ea7-da30-4139-affe-55db650564e8"). InnerVolumeSpecName "kube-api-access-r9cpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:27:36.697785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.697746 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4815ea7-da30-4139-affe-55db650564e8-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:36.697785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.697778 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9cpb\" (UniqueName: \"kubernetes.io/projected/b4815ea7-da30-4139-affe-55db650564e8-kube-api-access-r9cpb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:36.983726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:36.983646 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32052f15-57b6-41f5-981c-297a45b04a21" path="/var/lib/kubelet/pods/32052f15-57b6-41f5-981c-297a45b04a21/volumes" Nov 03 04:27:37.446466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:37.446438 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9" Nov 03 04:27:37.613093 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:37.613056 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:27:37.662248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:37.662203 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-246eeedd-5db678f74c-kmwz9"] Nov 03 04:27:38.983968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:38.983935 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4815ea7-da30-4139-affe-55db650564e8" path="/var/lib/kubelet/pods/b4815ea7-da30-4139-affe-55db650564e8/volumes" Nov 03 04:27:41.331757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:41.331710 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r"] Nov 03 04:27:41.336306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:41.336274 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-246eeedd-f222r"] Nov 03 04:27:42.982753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:42.982716 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef629c93-86ad-4508-9eac-653154172bfa" path="/var/lib/kubelet/pods/ef629c93-86ad-4508-9eac-653154172bfa/volumes" Nov 03 04:27:45.051159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:45.051123 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4"] Nov 03 04:27:45.100913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:45.100876 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56a73db-l67h4"] Nov 03 04:27:46.983245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:46.983210 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949a6d65-f8a6-4cd7-b495-369703bbf310" path="/var/lib/kubelet/pods/949a6d65-f8a6-4cd7-b495-369703bbf310/volumes" Nov 03 04:27:49.048368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:49.048333 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb"] Nov 03 04:27:49.097799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:49.097768 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3fca9-clcmb"] Nov 03 04:27:50.983628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:50.983590 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07cb0df3-e754-418e-8091-47e99fd4a5a3" path="/var/lib/kubelet/pods/07cb0df3-e754-418e-8091-47e99fd4a5a3/volumes" Nov 03 04:27:54.652949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:54.652914 5275 scope.go:122] "RemoveContainer" containerID="a0dbf8624f2f1ba4aac74f6ab0ed0436c3d0dea66d304995ac19609dfe9ed7d3" Nov 03 04:27:54.662323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:27:54.662302 5275 scope.go:122] "RemoveContainer" containerID="f14f1bd3801dde331223f1548612191bf588cb4de25c2035820a359ae131f7d8" Nov 03 04:28:02.329138 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:28:02.329095 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:28:08.076792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.076756 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb"] Nov 03 04:28:08.077385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077364 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ef629c93-86ad-4508-9eac-653154172bfa" containerName="revoke-keys" Nov 03 04:28:08.077478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077389 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef629c93-86ad-4508-9eac-653154172bfa" containerName="revoke-keys" Nov 03 04:28:08.077478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077421 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b4815ea7-da30-4139-affe-55db650564e8" containerName="payload-pre-processing" Nov 03 04:28:08.077478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077429 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4815ea7-da30-4139-affe-55db650564e8" containerName="payload-pre-processing" Nov 03 04:28:08.077650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077543 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4815ea7-da30-4139-affe-55db650564e8" containerName="payload-pre-processing" Nov 03 04:28:08.077650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.077565 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef629c93-86ad-4508-9eac-653154172bfa" containerName="revoke-keys" Nov 03 04:28:08.080745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.080723 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219296 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219339 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b572c1-6ea2-4a48-9633-41d7b8966db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219368 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8s7l\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-kube-api-access-w8s7l\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219524 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219569 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219660 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219687 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219826 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.219967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.219912 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.295820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.295785 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb"] Nov 03 04:28:08.320553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320513 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.320762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320573 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.320762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320606 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b572c1-6ea2-4a48-9633-41d7b8966db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.320762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320649 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8s7l\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-kube-api-access-w8s7l\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.320762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320698 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.320762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320725 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320778 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320802 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.320876 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.321093 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.321207 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.321274 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.321475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.321455 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.322323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.322297 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b572c1-6ea2-4a48-9633-41d7b8966db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.323518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.323492 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.323723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.323707 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.387575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.387507 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.387722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.387661 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8s7l\" (UniqueName: \"kubernetes.io/projected/73b572c1-6ea2-4a48-9633-41d7b8966db8-kube-api-access-w8s7l\") pod \"maas-default-gateway-openshift-default-76f4876667-vbhnb\" (UID: \"73b572c1-6ea2-4a48-9633-41d7b8966db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.398556 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.398529 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:08.829622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.829596 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb"] Nov 03 04:28:08.831812 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:28:08.831784 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b572c1_6ea2_4a48_9633_41d7b8966db8.slice/crio-5b61e647b52e612c6bccc4816c5cbdf142d5c82d3723f45c1a2edadfa21f023f WatchSource:0}: Error finding container 5b61e647b52e612c6bccc4816c5cbdf142d5c82d3723f45c1a2edadfa21f023f: Status 404 returned error can't find the container with id 5b61e647b52e612c6bccc4816c5cbdf142d5c82d3723f45c1a2edadfa21f023f Nov 03 04:28:08.834015 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.833983 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:28:08.834107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.834050 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:28:08.834107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:08.834082 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:28:09.636108 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:09.636069 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" event={"ID":"73b572c1-6ea2-4a48-9633-41d7b8966db8","Type":"ContainerStarted","Data":"a547406fd75df7bbc0e4e8765a4eec8bdebee59cca319275af250dc12626f70d"} Nov 03 04:28:09.636528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:09.636132 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" event={"ID":"73b572c1-6ea2-4a48-9633-41d7b8966db8","Type":"ContainerStarted","Data":"5b61e647b52e612c6bccc4816c5cbdf142d5c82d3723f45c1a2edadfa21f023f"} Nov 03 04:28:09.839776 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:09.839712 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podStartSLOduration=1.83969686 podStartE2EDuration="1.83969686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:09.805974758 +0000 UTC m=+5180.958663179" watchObservedRunningTime="2026-11-03 04:28:09.83969686 +0000 UTC m=+5180.992385277" Nov 03 04:28:10.399105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:10.399066 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:10.400681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:10.400649 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:10.400812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:10.400699 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:11.399898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:11.399841 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:11.400319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:11.399921 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:12.399919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:12.399881 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:12.400348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:12.399938 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:13.399959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:13.399918 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:13.400387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:13.399992 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:14.399382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.399346 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:14.399694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.399400 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:14.585134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.585102 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj"] Nov 03 04:28:14.588968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.588940 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.644715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.644682 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:28:14.699681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.699578 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:28:14.699889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.699764 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:28:14.699889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.699797 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:28:14.702139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.702116 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj"] Nov 03 04:28:14.710511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.710484 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.710668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.710519 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.710745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.710704 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cmt\" (UniqueName: \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.811804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.811765 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cmt\" (UniqueName: \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.812048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.811905 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.812048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.811939 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.812602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.812580 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.814651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.814622 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.875262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.875228 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cmt\" (UniqueName: \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\") pod \"maas-api-revoke-sub-85237e45-mk7kj\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:14.902009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:14.901976 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:15.289143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.289109 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj"] Nov 03 04:28:15.290261 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:28:15.290228 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23514697_397f_419f_9b6f_5107e67fe716.slice/crio-eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0 WatchSource:0}: Error finding container eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0: Status 404 returned error can't find the container with id eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0 Nov 03 04:28:15.399496 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.399450 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:15.399696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.399525 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:15.695117 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.695077 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" event={"ID":"23514697-397f-419f-9b6f-5107e67fe716","Type":"ContainerStarted","Data":"8335bb8553f7d78a0d2ef204b40f55544860b06760f3cb62ccfc00a06951f686"} Nov 03 04:28:15.695621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.695142 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" event={"ID":"23514697-397f-419f-9b6f-5107e67fe716","Type":"ContainerStarted","Data":"eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0"} Nov 03 04:28:15.812438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:15.812332 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" podStartSLOduration=1.8123162750000001 podStartE2EDuration="1.812316275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:15.810868797 +0000 UTC m=+5186.963557216" watchObservedRunningTime="2026-11-03 04:28:15.812316275 +0000 UTC m=+5186.965004696" Nov 03 04:28:16.399641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:16.399605 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" start-of-body= Nov 03 04:28:16.399842 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:16.399661 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": dial tcp 10.133.1.83:15021: connect: connection refused" Nov 03 04:28:16.714114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:16.714013 5275 generic.go:356] "Generic (PLEG): container finished" podID="23514697-397f-419f-9b6f-5107e67fe716" containerID="8335bb8553f7d78a0d2ef204b40f55544860b06760f3cb62ccfc00a06951f686" exitCode=0 Nov 03 04:28:16.714890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:16.714072 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" event={"ID":"23514697-397f-419f-9b6f-5107e67fe716","Type":"ContainerDied","Data":"8335bb8553f7d78a0d2ef204b40f55544860b06760f3cb62ccfc00a06951f686"} Nov 03 04:28:17.405265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.405226 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:17.405696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.405649 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:17.406431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.406411 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" Nov 03 04:28:17.859103 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.859081 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:17.939665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.939631 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\") pod \"23514697-397f-419f-9b6f-5107e67fe716\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " Nov 03 04:28:17.939891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.939721 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\") pod \"23514697-397f-419f-9b6f-5107e67fe716\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " Nov 03 04:28:17.939891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.939767 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\" (UniqueName: \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\") pod \"23514697-397f-419f-9b6f-5107e67fe716\" (UID: \"23514697-397f-419f-9b6f-5107e67fe716\") " Nov 03 04:28:17.940034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.940013 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca" pod "23514697-397f-419f-9b6f-5107e67fe716" (UID: "23514697-397f-419f-9b6f-5107e67fe716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:28:17.940142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.940116 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23514697-397f-419f-9b6f-5107e67fe716-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:28:17.942080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.942057 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt" pod "23514697-397f-419f-9b6f-5107e67fe716" (UID: "23514697-397f-419f-9b6f-5107e67fe716"). InnerVolumeSpecName "kube-api-access-b8cmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:28:17.942080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:17.942064 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca" pod "23514697-397f-419f-9b6f-5107e67fe716" (UID: "23514697-397f-419f-9b6f-5107e67fe716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:28:18.041348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:18.041312 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23514697-397f-419f-9b6f-5107e67fe716-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:28:18.041348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:18.041342 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8cmt\" (UniqueName: \"kubernetes.io/projected/23514697-397f-419f-9b6f-5107e67fe716-kube-api-access-b8cmt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:28:18.728074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:18.728042 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" Nov 03 04:28:18.728268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:18.728072 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj" event={"ID":"23514697-397f-419f-9b6f-5107e67fe716","Type":"ContainerDied","Data":"eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0"} Nov 03 04:28:18.728268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:18.728123 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeaf1f846371a0982dcc4dd53719dd42ba1de715464b4a59eed550d6bcf5a3f0" Nov 03 04:28:32.329758 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:28:32.329721 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:28:54.690425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:54.690345 5275 scope.go:122] "RemoveContainer" containerID="f89efe19b7c90c0dca2381379486f28ef330b006c047f3a26343e19deaf98e54" Nov 03 04:28:54.700012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:54.699992 5275 scope.go:122] "RemoveContainer" containerID="5e23cc848ce84f2915e14b108db90f04f934a923f28ec6ac9d9b0239f0506a1c" Nov 03 04:28:54.708816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:54.708796 5275 scope.go:122] "RemoveContainer" containerID="df04397437782fea693bae1f0ff984d0d7d89e41def936fd98166ab4f77807d7" Nov 03 04:28:54.717326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:28:54.717294 5275 scope.go:122] "RemoveContainer" containerID="39aa4e3d63e49490791f7ee3d28909aa6da2db89b006efdabb4549a5b93b2e7d" Nov 03 04:29:01.053180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.053139 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb"] Nov 03 04:29:01.053721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.053613 5275 cpu_manager.go:414] "containerMap: removing container" podUID="23514697-397f-419f-9b6f-5107e67fe716" containerName="revoke-keys" Nov 03 04:29:01.053721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.053626 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23514697-397f-419f-9b6f-5107e67fe716" containerName="revoke-keys" Nov 03 04:29:01.053864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.053731 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="23514697-397f-419f-9b6f-5107e67fe716" containerName="revoke-keys" Nov 03 04:29:01.056960 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.056936 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.093311 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.093275 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.093484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.093317 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6tm\" (UniqueName: \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.093484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.093437 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.114597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.114557 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb"] Nov 03 04:29:01.160565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.160530 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:29:01.160765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.160667 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:29:01.160765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.160734 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:29:01.160765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.160748 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:29:01.194569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.194527 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.194569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.194574 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6tm\" (UniqueName: \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.194815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.194621 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.195306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.195285 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.197665 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.197632 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.255573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.255535 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6tm\" (UniqueName: \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\") pod \"maas-api-revoke-sub-f904bff9-8kvlb\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.369948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.369866 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:01.750642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.750612 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb"] Nov 03 04:29:01.752273 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:29:01.752244 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21fa1a0_8072_4db6_87e9_0aedd4840079.slice/crio-31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4 WatchSource:0}: Error finding container 31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4: Status 404 returned error can't find the container with id 31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4 Nov 03 04:29:01.976915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:01.976878 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" event={"ID":"a21fa1a0-8072-4db6-87e9-0aedd4840079","Type":"ContainerStarted","Data":"31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4"} Nov 03 04:29:02.332274 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:29:02.332186 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:29:02.991327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:02.991233 5275 generic.go:356] "Generic (PLEG): container finished" podID="a21fa1a0-8072-4db6-87e9-0aedd4840079" containerID="5840d055660af59e1c56cc85286b6503abee2565b2c27532b4aad82c8b36d343" exitCode=0 Nov 03 04:29:02.991327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:02.991305 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" event={"ID":"a21fa1a0-8072-4db6-87e9-0aedd4840079","Type":"ContainerDied","Data":"5840d055660af59e1c56cc85286b6503abee2565b2c27532b4aad82c8b36d343"} Nov 03 04:29:04.133143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.133116 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:04.225579 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.225543 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\") pod \"a21fa1a0-8072-4db6-87e9-0aedd4840079\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " Nov 03 04:29:04.225766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.225604 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\" (UniqueName: \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\") pod \"a21fa1a0-8072-4db6-87e9-0aedd4840079\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " Nov 03 04:29:04.225766 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.225726 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\") pod \"a21fa1a0-8072-4db6-87e9-0aedd4840079\" (UID: \"a21fa1a0-8072-4db6-87e9-0aedd4840079\") " Nov 03 04:29:04.226174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.226147 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca" pod "a21fa1a0-8072-4db6-87e9-0aedd4840079" (UID: "a21fa1a0-8072-4db6-87e9-0aedd4840079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:29:04.227880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.227835 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca" pod "a21fa1a0-8072-4db6-87e9-0aedd4840079" (UID: "a21fa1a0-8072-4db6-87e9-0aedd4840079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:29:04.227974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.227959 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm" pod "a21fa1a0-8072-4db6-87e9-0aedd4840079" (UID: "a21fa1a0-8072-4db6-87e9-0aedd4840079"). InnerVolumeSpecName "kube-api-access-5n6tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:29:04.327577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.327541 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6tm\" (UniqueName: \"kubernetes.io/projected/a21fa1a0-8072-4db6-87e9-0aedd4840079-kube-api-access-5n6tm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:04.327577 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.327575 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:04.327820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:04.327591 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21fa1a0-8072-4db6-87e9-0aedd4840079-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:05.003221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:05.003193 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" Nov 03 04:29:05.003366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:05.003226 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb" event={"ID":"a21fa1a0-8072-4db6-87e9-0aedd4840079","Type":"ContainerDied","Data":"31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4"} Nov 03 04:29:05.003366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:05.003272 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ac53c6d127395191ebe6567767ea9a9534df1ed514e1b72d0043f24a1b6ef4" Nov 03 04:29:08.062601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.062565 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7"] Nov 03 04:29:08.063011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.062979 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a21fa1a0-8072-4db6-87e9-0aedd4840079" containerName="revoke-keys" Nov 03 04:29:08.063011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.062991 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21fa1a0-8072-4db6-87e9-0aedd4840079" containerName="revoke-keys" Nov 03 04:29:08.063110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.063076 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21fa1a0-8072-4db6-87e9-0aedd4840079" containerName="revoke-keys" Nov 03 04:29:08.066318 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.066297 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.160829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.160792 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7"] Nov 03 04:29:08.166157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166060 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519d3784-84ae-4be3-9404-536cc811d02a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166203 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166234 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166337 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166366 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166386 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjhw\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-kube-api-access-bqjhw\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166508 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166533 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519d3784-84ae-4be3-9404-536cc811d02a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.166664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.166651 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267286 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267330 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267330 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519d3784-84ae-4be3-9404-536cc811d02a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267366 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267396 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519d3784-84ae-4be3-9404-536cc811d02a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267416 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267438 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267481 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267507 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.267609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267536 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjhw\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-kube-api-access-bqjhw\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.268075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267811 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.268075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267886 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.268075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.267958 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.268398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.268149 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.268398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.268275 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519d3784-84ae-4be3-9404-536cc811d02a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.270018 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.269995 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519d3784-84ae-4be3-9404-536cc811d02a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.270245 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.270226 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519d3784-84ae-4be3-9404-536cc811d02a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.329104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.329029 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.329323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.329301 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjhw\" (UniqueName: \"kubernetes.io/projected/519d3784-84ae-4be3-9404-536cc811d02a-kube-api-access-bqjhw\") pod \"maas-default-gateway-openshift-default-76f4876667-fl4d7\" (UID: \"519d3784-84ae-4be3-9404-536cc811d02a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.378899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.378831 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:08.811175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.811147 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7"] Nov 03 04:29:08.812804 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:29:08.812776 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519d3784_84ae_4be3_9404_536cc811d02a.slice/crio-969c91a79b7ed73f687211375f439744fdfa2d7081cfc3f05286a6ce0d068dba WatchSource:0}: Error finding container 969c91a79b7ed73f687211375f439744fdfa2d7081cfc3f05286a6ce0d068dba: Status 404 returned error can't find the container with id 969c91a79b7ed73f687211375f439744fdfa2d7081cfc3f05286a6ce0d068dba Nov 03 04:29:08.816071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.816013 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:29:08.816184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.816169 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:29:08.816274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:08.816255 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:29:09.028146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.028105 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" event={"ID":"519d3784-84ae-4be3-9404-536cc811d02a","Type":"ContainerStarted","Data":"bb12eef7e0a538e81337ec40de16f6d35bf1d7be82dbc023ac324ac68ffb9626"} Nov 03 04:29:09.028332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.028168 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" event={"ID":"519d3784-84ae-4be3-9404-536cc811d02a","Type":"ContainerStarted","Data":"969c91a79b7ed73f687211375f439744fdfa2d7081cfc3f05286a6ce0d068dba"} Nov 03 04:29:09.150938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.150808 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podStartSLOduration=1.150793687 podStartE2EDuration="1.150793687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:29:09.149164742 +0000 UTC m=+5240.301853164" watchObservedRunningTime="2026-11-03 04:29:09.150793687 +0000 UTC m=+5240.303482108" Nov 03 04:29:09.379059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.379021 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:09.380704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.380674 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:09.380873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:09.380724 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:10.379316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:10.379273 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:10.379737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:10.379336 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:11.379995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:11.379958 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:11.380422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:11.380014 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:12.379675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:12.379639 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:12.379914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:12.379696 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:13.380194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:13.380146 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:13.380624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:13.380217 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:14.380321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:14.380278 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:14.380740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:14.380344 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:15.050650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:15.050617 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt"] Nov 03 04:29:15.102774 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:15.102723 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23dc2b5e-8cxxt"] Nov 03 04:29:15.379491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:15.379401 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body= Nov 03 04:29:15.379491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:15.379455 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" Nov 03 04:29:16.985477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:16.985432 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dad3d6-bf07-4181-b669-43b687c345e8" path="/var/lib/kubelet/pods/59dad3d6-bf07-4181-b669-43b687c345e8/volumes" Nov 03 04:29:17.380549 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:17.380509 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:29:17.380777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:17.380559 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:29:18.422026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:18.421981 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:29:18.422502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:18.422049 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:29:19.413635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:19.413606 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:19.414060 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:19.414041 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:19.414787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:19.414768 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" Nov 03 04:29:25.048031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:25.047997 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z"] Nov 03 04:29:25.097382 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:25.097339 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0741006-5zv2z"] Nov 03 04:29:26.982831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:26.982784 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c41fb6-f877-445f-a704-8217d002ef86" path="/var/lib/kubelet/pods/76c41fb6-f877-445f-a704-8217d002ef86/volumes" Nov 03 04:29:32.526873 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:29:32.525098 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:29:34.047983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:34.047945 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt"] Nov 03 04:29:34.097240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:34.097206 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358da859-669vt"] Nov 03 04:29:34.983396 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:34.983362 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e924eba-2a2a-4829-a342-74262e1a2bf1" path="/var/lib/kubelet/pods/0e924eba-2a2a-4829-a342-74262e1a2bf1/volumes" Nov 03 04:29:36.623799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.623759 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj"] Nov 03 04:29:36.628537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.628517 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.685620 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.685587 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj"] Nov 03 04:29:36.702204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.702171 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.702386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.702285 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.702386 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.702327 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffl6v\" (UniqueName: \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.732301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.732270 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:29:36.732301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.732270 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:29:36.732301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.732278 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:29:36.732558 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.732338 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:29:36.803983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.803946 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.804212 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.804025 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.804212 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.804072 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffl6v\" (UniqueName: \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.804713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.804684 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.806608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.806588 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.865120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.865082 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffl6v\" (UniqueName: \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\") pod \"maas-api-revoke-sub-f03629db-847xj\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:36.942544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:36.942451 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:37.320347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:37.320318 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj"] Nov 03 04:29:37.322289 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:29:37.322260 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0117e47a_8e54_4d2d_970a_5b710636bc69.slice/crio-c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5 WatchSource:0}: Error finding container c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5: Status 404 returned error can't find the container with id c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5 Nov 03 04:29:38.313387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:38.313341 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" event={"ID":"0117e47a-8e54-4d2d-970a-5b710636bc69","Type":"ContainerStarted","Data":"7de017fb1698c774667a3e1ecbb9bef07c33807023993b3cadf021660e245e99"} Nov 03 04:29:38.313830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:38.313412 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" event={"ID":"0117e47a-8e54-4d2d-970a-5b710636bc69","Type":"ContainerStarted","Data":"c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5"} Nov 03 04:29:38.431319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:38.431266 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" podStartSLOduration=2.431251556 podStartE2EDuration="2.431251556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:29:38.429530705 +0000 UTC m=+5269.582219125" watchObservedRunningTime="2026-11-03 04:29:38.431251556 +0000 UTC m=+5269.583939976" Nov 03 04:29:39.320841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:39.320811 5275 generic.go:356] "Generic (PLEG): container finished" podID="0117e47a-8e54-4d2d-970a-5b710636bc69" containerID="7de017fb1698c774667a3e1ecbb9bef07c33807023993b3cadf021660e245e99" exitCode=0 Nov 03 04:29:39.321295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:39.320884 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" event={"ID":"0117e47a-8e54-4d2d-970a-5b710636bc69","Type":"ContainerDied","Data":"7de017fb1698c774667a3e1ecbb9bef07c33807023993b3cadf021660e245e99"} Nov 03 04:29:40.461066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.461035 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:40.544246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.544199 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\" (UniqueName: \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\") pod \"0117e47a-8e54-4d2d-970a-5b710636bc69\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " Nov 03 04:29:40.544458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.544289 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\") pod \"0117e47a-8e54-4d2d-970a-5b710636bc69\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " Nov 03 04:29:40.544458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.544339 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\") pod \"0117e47a-8e54-4d2d-970a-5b710636bc69\" (UID: \"0117e47a-8e54-4d2d-970a-5b710636bc69\") " Nov 03 04:29:40.544685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.544656 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca" pod "0117e47a-8e54-4d2d-970a-5b710636bc69" (UID: "0117e47a-8e54-4d2d-970a-5b710636bc69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:29:40.546606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.546576 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca" pod "0117e47a-8e54-4d2d-970a-5b710636bc69" (UID: "0117e47a-8e54-4d2d-970a-5b710636bc69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:29:40.546679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.546619 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v" pod "0117e47a-8e54-4d2d-970a-5b710636bc69" (UID: "0117e47a-8e54-4d2d-970a-5b710636bc69"). InnerVolumeSpecName "kube-api-access-ffl6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:29:40.646141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.646044 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffl6v\" (UniqueName: \"kubernetes.io/projected/0117e47a-8e54-4d2d-970a-5b710636bc69-kube-api-access-ffl6v\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:40.646141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.646088 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:40.646141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:40.646099 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0117e47a-8e54-4d2d-970a-5b710636bc69-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:29:41.334075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:41.334043 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" Nov 03 04:29:41.334075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:41.334067 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj" event={"ID":"0117e47a-8e54-4d2d-970a-5b710636bc69","Type":"ContainerDied","Data":"c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5"} Nov 03 04:29:41.334335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:41.334116 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e5ae8eec9946e37ebe54d1408e1fb63b72d3e5e4bfc6bb30373562075739c5" Nov 03 04:29:51.052403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:51.052363 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld"] Nov 03 04:29:51.102110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:51.102073 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44c907-c7mld"] Nov 03 04:29:52.983571 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:52.983534 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bea7ac5-92cb-4665-be7c-95ec0b76c8e9" path="/var/lib/kubelet/pods/6bea7ac5-92cb-4665-be7c-95ec0b76c8e9/volumes" Nov 03 04:29:57.284315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.284281 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs"] Nov 03 04:29:57.284706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.284673 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0117e47a-8e54-4d2d-970a-5b710636bc69" containerName="revoke-keys" Nov 03 04:29:57.284706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.284683 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0117e47a-8e54-4d2d-970a-5b710636bc69" containerName="revoke-keys" Nov 03 04:29:57.284801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.284792 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0117e47a-8e54-4d2d-970a-5b710636bc69" containerName="revoke-keys" Nov 03 04:29:57.287836 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.287819 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.378671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.378637 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.378883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.378713 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.378883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.378860 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wfr\" (UniqueName: \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.389658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.389622 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:29:57.389658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.389640 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:29:57.389911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.389698 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:29:57.389911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.389764 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:29:57.391977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.391956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs"] Nov 03 04:29:57.480305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.480267 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.480513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.480356 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.480513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.480480 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wfr\" (UniqueName: \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.481295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.481253 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.483165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.483144 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.541808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.541737 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wfr\" (UniqueName: \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\") pod \"maas-api-revoke-sub-0165d7d2-bvgxs\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.600890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.600823 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:29:57.977880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:57.977775 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs"] Nov 03 04:29:57.981296 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:29:57.981271 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5713452_6373_4c43_bc5d_f7c27662789a.slice/crio-4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047 WatchSource:0}: Error finding container 4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047: Status 404 returned error can't find the container with id 4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047 Nov 03 04:29:58.493215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:58.493178 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" event={"ID":"c5713452-6373-4c43-bc5d-f7c27662789a","Type":"ContainerStarted","Data":"78cd632c484ffef3636d7fefbab58b0bd5a01a9d6e1bb66ab8f444d97e1cdf39"} Nov 03 04:29:58.493591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:58.493234 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" event={"ID":"c5713452-6373-4c43-bc5d-f7c27662789a","Type":"ContainerStarted","Data":"4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047"} Nov 03 04:29:58.610241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:58.610178 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" podStartSLOduration=1.6101619980000001 podStartE2EDuration="1.610161998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:29:58.608753673 +0000 UTC m=+5289.761442094" watchObservedRunningTime="2026-11-03 04:29:58.610161998 +0000 UTC m=+5289.762850418" Nov 03 04:29:59.500516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:59.500478 5275 generic.go:356] "Generic (PLEG): container finished" podID="c5713452-6373-4c43-bc5d-f7c27662789a" containerID="78cd632c484ffef3636d7fefbab58b0bd5a01a9d6e1bb66ab8f444d97e1cdf39" exitCode=0 Nov 03 04:29:59.500976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:29:59.500572 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" event={"ID":"c5713452-6373-4c43-bc5d-f7c27662789a","Type":"ContainerDied","Data":"78cd632c484ffef3636d7fefbab58b0bd5a01a9d6e1bb66ab8f444d97e1cdf39"} Nov 03 04:30:00.648921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.648893 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:30:00.709787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.709746 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\") pod \"c5713452-6373-4c43-bc5d-f7c27662789a\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " Nov 03 04:30:00.710026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.709860 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\" (UniqueName: \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\") pod \"c5713452-6373-4c43-bc5d-f7c27662789a\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " Nov 03 04:30:00.710026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.710005 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\") pod \"c5713452-6373-4c43-bc5d-f7c27662789a\" (UID: \"c5713452-6373-4c43-bc5d-f7c27662789a\") " Nov 03 04:30:00.710176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.710145 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca" pod "c5713452-6373-4c43-bc5d-f7c27662789a" (UID: "c5713452-6373-4c43-bc5d-f7c27662789a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:30:00.710346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.710329 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:00.712225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.712195 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca" pod "c5713452-6373-4c43-bc5d-f7c27662789a" (UID: "c5713452-6373-4c43-bc5d-f7c27662789a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:30:00.712225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.712202 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr" pod "c5713452-6373-4c43-bc5d-f7c27662789a" (UID: "c5713452-6373-4c43-bc5d-f7c27662789a"). InnerVolumeSpecName "kube-api-access-96wfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:30:00.810890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.810836 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wfr\" (UniqueName: \"kubernetes.io/projected/c5713452-6373-4c43-bc5d-f7c27662789a-kube-api-access-96wfr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:00.810890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:00.810885 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5713452-6373-4c43-bc5d-f7c27662789a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:01.513590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:01.513551 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" event={"ID":"c5713452-6373-4c43-bc5d-f7c27662789a","Type":"ContainerDied","Data":"4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047"} Nov 03 04:30:01.513590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:01.513574 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs" Nov 03 04:30:01.513886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:01.513606 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4433cd8f6b427109cc0a460374a4eb66395d7612636796385fccf23c8822d047" Nov 03 04:30:02.329153 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:30:02.329113 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:30:09.291090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.291051 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4"] Nov 03 04:30:09.291559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.291473 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c5713452-6373-4c43-bc5d-f7c27662789a" containerName="revoke-keys" Nov 03 04:30:09.291559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.291485 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5713452-6373-4c43-bc5d-f7c27662789a" containerName="revoke-keys" Nov 03 04:30:09.291651 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.291561 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5713452-6373-4c43-bc5d-f7c27662789a" containerName="revoke-keys" Nov 03 04:30:09.295090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.295068 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.349784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.349749 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:30:09.350135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.349897 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:30:09.399679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.399647 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:30:09.399679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.399648 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:30:09.402013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.401983 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4"] Nov 03 04:30:09.402149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.402123 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.402244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.402230 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.402285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.402255 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qb6\" (UniqueName: \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.503836 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.503800 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.503836 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.503838 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qb6\" (UniqueName: \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.504146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.503960 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.504500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.504479 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.506398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.506375 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.565110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.565029 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qb6\" (UniqueName: \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\") pod \"maas-api-revoke-sub-6b19338f-xlxs4\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.608142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.608102 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:09.987422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:09.987393 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4"] Nov 03 04:30:09.989390 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:30:09.989360 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c9e375_7fea_4678_8eb1_7e6a3929392d.slice/crio-08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90 WatchSource:0}: Error finding container 08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90: Status 404 returned error can't find the container with id 08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90 Nov 03 04:30:10.567998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:10.567962 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" event={"ID":"66c9e375-7fea-4678-8eb1-7e6a3929392d","Type":"ContainerStarted","Data":"c193e9eef36112ccf89475ac0809bed9b9e2183a24fd3ddd307f123bc8e9a7e9"} Nov 03 04:30:10.568486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:10.568060 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" event={"ID":"66c9e375-7fea-4678-8eb1-7e6a3929392d","Type":"ContainerStarted","Data":"08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90"} Nov 03 04:30:10.687839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:10.687685 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" podStartSLOduration=1.687662763 podStartE2EDuration="1.687662763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:30:10.68400455 +0000 UTC m=+5301.836692971" watchObservedRunningTime="2026-11-03 04:30:10.687662763 +0000 UTC m=+5301.840351185" Nov 03 04:30:11.575652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:11.575614 5275 generic.go:356] "Generic (PLEG): container finished" podID="66c9e375-7fea-4678-8eb1-7e6a3929392d" containerID="c193e9eef36112ccf89475ac0809bed9b9e2183a24fd3ddd307f123bc8e9a7e9" exitCode=0 Nov 03 04:30:11.576122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:11.575709 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" event={"ID":"66c9e375-7fea-4678-8eb1-7e6a3929392d","Type":"ContainerDied","Data":"c193e9eef36112ccf89475ac0809bed9b9e2183a24fd3ddd307f123bc8e9a7e9"} Nov 03 04:30:12.720130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.720103 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:12.839416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.839324 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\" (UniqueName: \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\") pod \"66c9e375-7fea-4678-8eb1-7e6a3929392d\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " Nov 03 04:30:12.839603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.839426 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\") pod \"66c9e375-7fea-4678-8eb1-7e6a3929392d\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " Nov 03 04:30:12.839603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.839476 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\") pod \"66c9e375-7fea-4678-8eb1-7e6a3929392d\" (UID: \"66c9e375-7fea-4678-8eb1-7e6a3929392d\") " Nov 03 04:30:12.840165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.840135 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca" pod "66c9e375-7fea-4678-8eb1-7e6a3929392d" (UID: "66c9e375-7fea-4678-8eb1-7e6a3929392d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:30:12.841903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.841873 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6" pod "66c9e375-7fea-4678-8eb1-7e6a3929392d" (UID: "66c9e375-7fea-4678-8eb1-7e6a3929392d"). InnerVolumeSpecName "kube-api-access-54qb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:30:12.841903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.841873 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca" pod "66c9e375-7fea-4678-8eb1-7e6a3929392d" (UID: "66c9e375-7fea-4678-8eb1-7e6a3929392d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:30:12.941065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.941023 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qb6\" (UniqueName: \"kubernetes.io/projected/66c9e375-7fea-4678-8eb1-7e6a3929392d-kube-api-access-54qb6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:12.941065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.941061 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:12.941065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:12.941071 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c9e375-7fea-4678-8eb1-7e6a3929392d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:13.588064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:13.588025 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" event={"ID":"66c9e375-7fea-4678-8eb1-7e6a3929392d","Type":"ContainerDied","Data":"08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90"} Nov 03 04:30:13.588064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:13.588044 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4" Nov 03 04:30:13.588293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:13.588085 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c04fc3d7c970b91f3b0bd860197255032841fef3a0b61c85c8eab4d8b8cf90" Nov 03 04:30:26.113060 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.112981 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:30:26.113490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.113469 5275 cpu_manager.go:414] "containerMap: removing container" podUID="66c9e375-7fea-4678-8eb1-7e6a3929392d" containerName="revoke-keys" Nov 03 04:30:26.113601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.113500 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c9e375-7fea-4678-8eb1-7e6a3929392d" containerName="revoke-keys" Nov 03 04:30:26.113680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.113632 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c9e375-7fea-4678-8eb1-7e6a3929392d" containerName="revoke-keys" Nov 03 04:30:26.117056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.117037 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.191664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.191627 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-edfabb23-epp-sa-dockercfg-g7jgt\"" Nov 03 04:30:26.222090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.222050 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-edfabb23-kserve-self-signed-certs\"" Nov 03 04:30:26.224591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.224563 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:30:26.235014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.234982 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.235206 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.235045 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxjz\" (UniqueName: \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.335767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.335718 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.335970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.335781 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxjz\" (UniqueName: \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.338438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.338404 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.396834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.396752 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxjz\" (UniqueName: \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\") pod \"e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.430218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.430184 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:26.809543 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.809511 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:30:26.810645 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:30:26.810622 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd782b86e_05cc_4a1b_8a9a_0e5c1fc4507a.slice/crio-3462df5a3cd27b187810b2cc8b158939beb6e3248a87ec5cad1eb49b428abe3b WatchSource:0}: Error finding container 3462df5a3cd27b187810b2cc8b158939beb6e3248a87ec5cad1eb49b428abe3b: Status 404 returned error can't find the container with id 3462df5a3cd27b187810b2cc8b158939beb6e3248a87ec5cad1eb49b428abe3b Nov 03 04:30:26.812699 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:26.812681 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:30:27.723325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:27.723289 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" event={"ID":"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a","Type":"ContainerStarted","Data":"3462df5a3cd27b187810b2cc8b158939beb6e3248a87ec5cad1eb49b428abe3b"} Nov 03 04:30:29.747510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:29.747421 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" event={"ID":"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a","Type":"ContainerStarted","Data":"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe"} Nov 03 04:30:29.748079 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:29.747534 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:30:29.866306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:29.866240 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podStartSLOduration=1.586661002 podStartE2EDuration="3.866224094s" totalImagesPullingTime="2.279563092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:30:29.863817206 +0000 UTC m=+5321.016505628" watchObservedRunningTime="2026-11-03 04:30:29.866224094 +0000 UTC m=+5321.018912515" Nov 03 04:30:32.461873 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:30:32.457113 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:30:33.452983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:33.452943 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:30:33.453315 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:33.453282 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:30:44.051281 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:44.051249 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4"] Nov 03 04:30:44.100402 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:44.100370 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-927a8430-6ngx4"] Nov 03 04:30:44.983671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:44.983637 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1a31d7-6ee9-4a16-98c0-14534585459b" path="/var/lib/kubelet/pods/8e1a31d7-6ee9-4a16-98c0-14534585459b/volumes" Nov 03 04:30:54.828490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:54.828448 5275 scope.go:122] "RemoveContainer" containerID="f16cd96736761c8cf35ca5a14c919f5a4cf2bcc20a737a7f402bcb8a494d56e1" Nov 03 04:30:54.837941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:54.837921 5275 scope.go:122] "RemoveContainer" containerID="ef97b2a3e7a592bf02285c4688f751d61e5406d6392a06da36951ff68268033a" Nov 03 04:30:54.847272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:54.847250 5275 scope.go:122] "RemoveContainer" containerID="3c9b190f38dbdd2e63e80d404761529e2dc66a25118e8a9a2ede4e5418ec6c14" Nov 03 04:30:54.856208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:30:54.856177 5275 scope.go:122] "RemoveContainer" containerID="9d58cabfebfc6ef2e60e45d39e9e33eb672e2880e77c256f052575b7869891a8" Nov 03 04:31:00.760358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:00.760324 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:31:01.051106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:01.051056 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg"] Nov 03 04:31:01.054595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:01.054569 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c9f3c9-x5nqg"] Nov 03 04:31:02.332229 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:31:02.332190 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:31:02.982789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:02.982759 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa9443e-2f73-41f4-9236-c409ff21b1e5" path="/var/lib/kubelet/pods/3aa9443e-2f73-41f4-9236-c409ff21b1e5/volumes" Nov 03 04:31:32.329718 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:31:32.329678 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:31:40.994536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:40.994504 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:31:50.530065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:50.530036 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:31:50.571400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:50.571372 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:31:50.760365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:50.760307 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:31:54.895681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:54.895603 5275 scope.go:122] "RemoveContainer" containerID="46699fee036493ce3a4275c42869457b227ff1b6d92bee3b80eb4c34a3775811" Nov 03 04:31:56.026644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:31:56.026587 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" containerID="cri-o://40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe" gracePeriod=45 Nov 03 04:32:00.760032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:00.759994 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:32:02.329962 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:32:02.329925 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:32:05.466886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:05.463969 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:32:05.466886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:05.464051 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:32:10.760188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:10.760143 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:32:10.761006 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:10.760259 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:32:20.759674 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:20.759623 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:32:26.276592 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.276559 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:32:26.400497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.400404 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\") pod \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " Nov 03 04:32:26.400689 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.400499 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\" (UniqueName: \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\") pod \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\" (UID: \"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a\") " Nov 03 04:32:26.402937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.402906 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs" pod "d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" (UID: "d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:32:26.403053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.402987 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz" pod "d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" (UID: "d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a"). InnerVolumeSpecName "kube-api-access-slxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:32:26.501383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.501352 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:32:26.501383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.501383 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxjz\" (UniqueName: \"kubernetes.io/projected/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a-kube-api-access-slxjz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:32:26.575879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.575813 5275 generic.go:356] "Generic (PLEG): container finished" podID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerID="40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe" exitCode=0 Nov 03 04:32:26.576094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.575897 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" Nov 03 04:32:26.576094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.575901 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" event={"ID":"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a","Type":"ContainerDied","Data":"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe"} Nov 03 04:32:26.576094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.575971 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx" event={"ID":"d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a","Type":"ContainerDied","Data":"3462df5a3cd27b187810b2cc8b158939beb6e3248a87ec5cad1eb49b428abe3b"} Nov 03 04:32:26.576094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.576001 5275 scope.go:122] "RemoveContainer" containerID="40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe" Nov 03 04:32:26.586007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.585986 5275 scope.go:122] "RemoveContainer" containerID="40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe" Nov 03 04:32:26.586314 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:32:26.586291 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe\": container with ID starting with 40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe not found: ID does not exist" containerID="40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe" Nov 03 04:32:26.586373 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.586325 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe"} err="failed to get container status \"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe\": rpc error: code = NotFound desc = could not find container \"40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe\": container with ID starting with 40b8a7129f26457cbb78df68c3ee19996bb4fc81958a14627955d8fdaeb7e9fe not found: ID does not exist" Nov 03 04:32:26.740885 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.740767 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:32:26.746518 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.746484 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-router-scheduler-7d699f75c9-lqqqx"] Nov 03 04:32:26.983520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:26.983483 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" path="/var/lib/kubelet/pods/d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a/volumes" Nov 03 04:32:32.332709 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:32:32.332668 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:32:54.918328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:54.918296 5275 scope.go:122] "RemoveContainer" containerID="3f2a4be9ac2647b25a129dbb2be925ded9f0604b63a807447ef7f72e9a00b5f0" Nov 03 04:32:54.927613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:32:54.927588 5275 scope.go:122] "RemoveContainer" containerID="de46f4d59c1304048b269176ad4556a74bf242a7a786e00fdb728a52f1a4f0ec" Nov 03 04:33:02.333882 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:33:02.333734 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:33:18.051658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.051598 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj"] Nov 03 04:33:18.101321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.101285 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mk7kj"] Nov 03 04:33:18.911275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.911237 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7"] Nov 03 04:33:18.911666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.911653 5275 cpu_manager.go:414] "containerMap: removing container" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" Nov 03 04:33:18.911753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.911667 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" Nov 03 04:33:18.911753 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.911739 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="d782b86e-05cc-4a1b-8a9a-0e5c1fc4507a" containerName="main" Nov 03 04:33:18.914987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.914968 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:18.968173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.968137 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:33:18.968173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.968161 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:33:18.991879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:18.986027 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23514697-397f-419f-9b6f-5107e67fe716" path="/var/lib/kubelet/pods/23514697-397f-419f-9b6f-5107e67fe716/volumes" Nov 03 04:33:19.018012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.017975 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:33:19.018241 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.018226 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:33:19.020364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.020342 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7"] Nov 03 04:33:19.035031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.035000 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.035183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.035038 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.035234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.035184 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrktz\" (UniqueName: \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.136564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.136502 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrktz\" (UniqueName: \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.137052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.136601 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.137052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.136625 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.137277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.137256 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.139253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.139230 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.198044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.197963 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrktz\" (UniqueName: \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\") pod \"maas-api-revoke-sub-2b4b3372-vttp7\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.237880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.237815 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:19.616710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.616683 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7"] Nov 03 04:33:19.618678 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:33:19.618650 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd298e5_6957_4db1_b4e6_58b15ff7712d.slice/crio-c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de WatchSource:0}: Error finding container c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de: Status 404 returned error can't find the container with id c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de Nov 03 04:33:19.886349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.886261 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" event={"ID":"1bd298e5-6957-4db1-b4e6-58b15ff7712d","Type":"ContainerStarted","Data":"16a2af183358e4e3c327b7514484f34b1dd1411def46c8c4d26b8ae16f6b8c09"} Nov 03 04:33:19.886349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:19.886313 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" event={"ID":"1bd298e5-6957-4db1-b4e6-58b15ff7712d","Type":"ContainerStarted","Data":"c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de"} Nov 03 04:33:20.004248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:20.004195 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" podStartSLOduration=2.004180323 podStartE2EDuration="2.004180323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:33:20.001664795 +0000 UTC m=+5491.154353217" watchObservedRunningTime="2026-11-03 04:33:20.004180323 +0000 UTC m=+5491.156868744" Nov 03 04:33:20.893959 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:20.893926 5275 generic.go:356] "Generic (PLEG): container finished" podID="1bd298e5-6957-4db1-b4e6-58b15ff7712d" containerID="16a2af183358e4e3c327b7514484f34b1dd1411def46c8c4d26b8ae16f6b8c09" exitCode=0 Nov 03 04:33:20.894383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:20.894020 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" event={"ID":"1bd298e5-6957-4db1-b4e6-58b15ff7712d","Type":"ContainerDied","Data":"16a2af183358e4e3c327b7514484f34b1dd1411def46c8c4d26b8ae16f6b8c09"} Nov 03 04:33:22.048326 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.048285 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:22.167449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.167342 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\" (UniqueName: \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\") pod \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " Nov 03 04:33:22.167449 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.167426 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\") pod \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " Nov 03 04:33:22.167714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.167479 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\") pod \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\" (UID: \"1bd298e5-6957-4db1-b4e6-58b15ff7712d\") " Nov 03 04:33:22.167827 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.167800 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca" pod "1bd298e5-6957-4db1-b4e6-58b15ff7712d" (UID: "1bd298e5-6957-4db1-b4e6-58b15ff7712d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:33:22.168019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.168002 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:22.169787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.169755 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz" pod "1bd298e5-6957-4db1-b4e6-58b15ff7712d" (UID: "1bd298e5-6957-4db1-b4e6-58b15ff7712d"). InnerVolumeSpecName "kube-api-access-qrktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:33:22.169787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.169775 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca" pod "1bd298e5-6957-4db1-b4e6-58b15ff7712d" (UID: "1bd298e5-6957-4db1-b4e6-58b15ff7712d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:33:22.269135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.269098 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrktz\" (UniqueName: \"kubernetes.io/projected/1bd298e5-6957-4db1-b4e6-58b15ff7712d-kube-api-access-qrktz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:22.269135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.269130 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd298e5-6957-4db1-b4e6-58b15ff7712d-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:22.910140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.910049 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" Nov 03 04:33:22.910140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.910040 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7" event={"ID":"1bd298e5-6957-4db1-b4e6-58b15ff7712d","Type":"ContainerDied","Data":"c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de"} Nov 03 04:33:22.910368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:22.910182 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6347fcf4e2e59334b64e1d3eebe21efe67a29070358ac8f1fb5a284981339de" Nov 03 04:33:32.330319 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:33:32.330281 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:33:54.956455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:33:54.956421 5275 scope.go:122] "RemoveContainer" containerID="7904215625f846b9f51a0682d3acc366c82384d5f6f9d04387416a722cd2a627" Nov 03 04:34:02.329091 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:34:02.329055 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:34:05.057923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:05.057888 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb"] Nov 03 04:34:05.072056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:05.072021 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8kvlb"] Nov 03 04:34:06.983500 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:06.983466 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21fa1a0-8072-4db6-87e9-0aedd4840079" path="/var/lib/kubelet/pods/a21fa1a0-8072-4db6-87e9-0aedd4840079/volumes" Nov 03 04:34:26.008970 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.008929 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs"] Nov 03 04:34:26.009394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.009352 5275 cpu_manager.go:414] "containerMap: removing container" podUID="1bd298e5-6957-4db1-b4e6-58b15ff7712d" containerName="revoke-keys" Nov 03 04:34:26.009394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.009363 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd298e5-6957-4db1-b4e6-58b15ff7712d" containerName="revoke-keys" Nov 03 04:34:26.009512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.009456 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd298e5-6957-4db1-b4e6-58b15ff7712d" containerName="revoke-keys" Nov 03 04:34:26.012763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.012743 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.068188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.068145 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:34:26.114447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.114414 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:34:26.114447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.114436 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:34:26.114447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.114440 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:34:26.116757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.116734 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs"] Nov 03 04:34:26.124145 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.124117 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.124268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.124246 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhwd\" (UniqueName: \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.124327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.124313 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.225088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.225039 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhwd\" (UniqueName: \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.225088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.225085 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.225372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.225154 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.225797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.225775 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.227715 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.227689 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.286466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.286436 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhwd\" (UniqueName: \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.326001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.325966 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:26.701656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:26.701614 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs"] Nov 03 04:34:26.704038 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:34:26.704009 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f368cb_4645_4c21_83ae_c19aeea70dc7.slice/crio-e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22 WatchSource:0}: Error finding container e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22: Status 404 returned error can't find the container with id e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22 Nov 03 04:34:27.268722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:27.268632 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" event={"ID":"62f368cb-4645-4c21-83ae-c19aeea70dc7","Type":"ContainerStarted","Data":"d18bd19980cc83c4e52f91a9d846ac8040f894d2b839c73d269a8229b780b435"} Nov 03 04:34:27.268722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:27.268686 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" event={"ID":"62f368cb-4645-4c21-83ae-c19aeea70dc7","Type":"ContainerStarted","Data":"e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22"} Nov 03 04:34:27.386012 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:27.385951 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" podStartSLOduration=2.3859368610000002 podStartE2EDuration="2.385936861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:34:27.384165273 +0000 UTC m=+5558.536853694" watchObservedRunningTime="2026-11-03 04:34:27.385936861 +0000 UTC m=+5558.538625282" Nov 03 04:34:28.275696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:28.275661 5275 generic.go:356] "Generic (PLEG): container finished" podID="62f368cb-4645-4c21-83ae-c19aeea70dc7" containerID="d18bd19980cc83c4e52f91a9d846ac8040f894d2b839c73d269a8229b780b435" exitCode=0 Nov 03 04:34:28.276165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:28.275755 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" event={"ID":"62f368cb-4645-4c21-83ae-c19aeea70dc7","Type":"ContainerDied","Data":"d18bd19980cc83c4e52f91a9d846ac8040f894d2b839c73d269a8229b780b435"} Nov 03 04:34:29.425211 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.425184 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:29.560767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.560685 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\" (UniqueName: \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\") pod \"62f368cb-4645-4c21-83ae-c19aeea70dc7\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " Nov 03 04:34:29.560948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.560832 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\") pod \"62f368cb-4645-4c21-83ae-c19aeea70dc7\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " Nov 03 04:34:29.560948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.560897 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\") pod \"62f368cb-4645-4c21-83ae-c19aeea70dc7\" (UID: \"62f368cb-4645-4c21-83ae-c19aeea70dc7\") " Nov 03 04:34:29.561204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.561166 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca" pod "62f368cb-4645-4c21-83ae-c19aeea70dc7" (UID: "62f368cb-4645-4c21-83ae-c19aeea70dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:34:29.563406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.563376 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd" pod "62f368cb-4645-4c21-83ae-c19aeea70dc7" (UID: "62f368cb-4645-4c21-83ae-c19aeea70dc7"). InnerVolumeSpecName "kube-api-access-zkhwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:34:29.563560 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.563426 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca" pod "62f368cb-4645-4c21-83ae-c19aeea70dc7" (UID: "62f368cb-4645-4c21-83ae-c19aeea70dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:34:29.662085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.662049 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:34:29.662085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.662079 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f368cb-4645-4c21-83ae-c19aeea70dc7-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:34:29.662085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:29.662089 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkhwd\" (UniqueName: \"kubernetes.io/projected/62f368cb-4645-4c21-83ae-c19aeea70dc7-kube-api-access-zkhwd\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:34:30.291178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:30.291142 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" Nov 03 04:34:30.291178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:30.291171 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs" event={"ID":"62f368cb-4645-4c21-83ae-c19aeea70dc7","Type":"ContainerDied","Data":"e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22"} Nov 03 04:34:30.291426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:30.291230 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3cb9475b9021859e867380ed5987760395d30ec2be8537e77c9343ee4152e22" Nov 03 04:34:32.329951 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:34:32.329913 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:34:41.056218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:41.056178 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj"] Nov 03 04:34:41.105120 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:41.105077 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-847xj"] Nov 03 04:34:42.983019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:42.982982 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0117e47a-8e54-4d2d-970a-5b710636bc69" path="/var/lib/kubelet/pods/0117e47a-8e54-4d2d-970a-5b710636bc69/volumes" Nov 03 04:34:45.572912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:45.572868 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs"] Nov 03 04:34:45.621809 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:45.621774 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-deca42-qh4bs"] Nov 03 04:34:46.983161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:46.983129 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f368cb-4645-4c21-83ae-c19aeea70dc7" path="/var/lib/kubelet/pods/62f368cb-4645-4c21-83ae-c19aeea70dc7/volumes" Nov 03 04:34:54.979468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:34:54.979384 5275 scope.go:122] "RemoveContainer" containerID="8335bb8553f7d78a0d2ef204b40f55544860b06760f3cb62ccfc00a06951f686" Nov 03 04:35:01.088494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:01.088450 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs"] Nov 03 04:35:01.109345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:01.109290 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bvgxs"] Nov 03 04:35:02.431430 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:35:02.431351 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:35:03.009348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:03.009303 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5713452-6373-4c43-bc5d-f7c27662789a" path="/var/lib/kubelet/pods/c5713452-6373-4c43-bc5d-f7c27662789a/volumes" Nov 03 04:35:03.460633 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:03.460593 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:35:03.461442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:03.461405 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:35:13.052912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:13.052872 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4"] Nov 03 04:35:13.101117 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:13.101083 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlxs4"] Nov 03 04:35:14.983034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:14.982994 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c9e375-7fea-4678-8eb1-7e6a3929392d" path="/var/lib/kubelet/pods/66c9e375-7fea-4678-8eb1-7e6a3929392d/volumes" Nov 03 04:35:21.581925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.581887 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:35:21.582337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.582323 5275 cpu_manager.go:414] "containerMap: removing container" podUID="62f368cb-4645-4c21-83ae-c19aeea70dc7" containerName="revoke-keys" Nov 03 04:35:21.582337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.582335 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f368cb-4645-4c21-83ae-c19aeea70dc7" containerName="revoke-keys" Nov 03 04:35:21.582418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.582398 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f368cb-4645-4c21-83ae-c19aeea70dc7" containerName="revoke-keys" Nov 03 04:35:21.585640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.585621 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672227 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672276 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672286 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672393 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672447 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htj22\" (UniqueName: \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672554 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.672597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.672577 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.687686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.687645 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-82587b78-kserve-self-signed-certs\"" Nov 03 04:35:21.689765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.689744 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:35:21.773671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773628 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.773671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773673 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.773984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773730 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.773984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773753 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.773984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773803 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.773984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.773891 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htj22\" (UniqueName: \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.774271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.774242 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.774411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.774292 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.774411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.774350 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.776275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.776257 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.776456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.776438 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.835419 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.835352 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htj22\" (UniqueName: \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\") pod \"e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:21.897607 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:21.897569 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:22.286042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:22.286016 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:35:22.288600 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:35:22.288573 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36087bc1_d41f_432c_b9d5_36dd3bea7fb1.slice/crio-154cfae17d92f9fcb1b38e414a40e5471684b1167e55a12f6075fe32ad2d04bb WatchSource:0}: Error finding container 154cfae17d92f9fcb1b38e414a40e5471684b1167e55a12f6075fe32ad2d04bb: Status 404 returned error can't find the container with id 154cfae17d92f9fcb1b38e414a40e5471684b1167e55a12f6075fe32ad2d04bb Nov 03 04:35:22.645533 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:22.645428 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" event={"ID":"36087bc1-d41f-432c-b9d5-36dd3bea7fb1","Type":"ContainerStarted","Data":"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453"} Nov 03 04:35:22.645533 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:22.645482 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" event={"ID":"36087bc1-d41f-432c-b9d5-36dd3bea7fb1","Type":"ContainerStarted","Data":"154cfae17d92f9fcb1b38e414a40e5471684b1167e55a12f6075fe32ad2d04bb"} Nov 03 04:35:22.763110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:22.763060 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" podStartSLOduration=1.763046294 podStartE2EDuration="1.763046294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:35:22.761927822 +0000 UTC m=+5613.914616240" watchObservedRunningTime="2026-11-03 04:35:22.763046294 +0000 UTC m=+5613.915734714" Nov 03 04:35:22.898624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:22.898539 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:31.898184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:31.898143 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:31.912480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:31.912455 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:32.329449 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:35:32.329410 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:35:32.742676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:32.742593 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:35:49.828168 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.828132 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj"] Nov 03 04:35:49.831789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.831770 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:49.888982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.888948 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj"] Nov 03 04:35:49.935388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.935343 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:35:49.935388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.935354 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:35:49.935388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.935385 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:35:49.935657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.935443 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:35:49.959982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.959942 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwp9\" (UniqueName: \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:49.959982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.959983 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:49.960204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:49.960155 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.061123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.061082 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.061307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.061158 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwp9\" (UniqueName: \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.061307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.061191 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.061986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.061957 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.063743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.063715 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.122443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.122368 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwp9\" (UniqueName: \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\") pod \"maas-api-revoke-sub-452b6d7b-t7qkj\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.145261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.145222 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:50.519903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.519875 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj"] Nov 03 04:35:50.522102 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:35:50.522074 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda6461f_36bc_445a_bc34_07298002e42e.slice/crio-9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a WatchSource:0}: Error finding container 9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a: Status 404 returned error can't find the container with id 9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a Nov 03 04:35:50.835596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:50.835566 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" event={"ID":"dda6461f-36bc-445a-bc34-07298002e42e","Type":"ContainerStarted","Data":"9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a"} Nov 03 04:35:51.846784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:51.846742 5275 generic.go:356] "Generic (PLEG): container finished" podID="dda6461f-36bc-445a-bc34-07298002e42e" containerID="3e51b4a098af387eb418ae0775621012c1ff3b17d1460c20f6a3fa937580b826" exitCode=0 Nov 03 04:35:51.847299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:51.846808 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" event={"ID":"dda6461f-36bc-445a-bc34-07298002e42e","Type":"ContainerDied","Data":"3e51b4a098af387eb418ae0775621012c1ff3b17d1460c20f6a3fa937580b826"} Nov 03 04:35:53.575942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.575641 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:53.666146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.665149 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\") pod \"dda6461f-36bc-445a-bc34-07298002e42e\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " Nov 03 04:35:53.666146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.665326 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\") pod \"dda6461f-36bc-445a-bc34-07298002e42e\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " Nov 03 04:35:53.666146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.665402 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\" (UniqueName: \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\") pod \"dda6461f-36bc-445a-bc34-07298002e42e\" (UID: \"dda6461f-36bc-445a-bc34-07298002e42e\") " Nov 03 04:35:53.666951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.666904 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca" pod "dda6461f-36bc-445a-bc34-07298002e42e" (UID: "dda6461f-36bc-445a-bc34-07298002e42e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:35:53.676717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.676675 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca" pod "dda6461f-36bc-445a-bc34-07298002e42e" (UID: "dda6461f-36bc-445a-bc34-07298002e42e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:35:53.677123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.677086 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9" pod "dda6461f-36bc-445a-bc34-07298002e42e" (UID: "dda6461f-36bc-445a-bc34-07298002e42e"). InnerVolumeSpecName "kube-api-access-hfwp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:35:53.771893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.767818 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:35:53.772216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.772180 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwp9\" (UniqueName: \"kubernetes.io/projected/dda6461f-36bc-445a-bc34-07298002e42e-kube-api-access-hfwp9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:35:53.772389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.772371 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda6461f-36bc-445a-bc34-07298002e42e-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:35:53.921053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.918782 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" event={"ID":"dda6461f-36bc-445a-bc34-07298002e42e","Type":"ContainerDied","Data":"9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a"} Nov 03 04:35:53.921053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.918913 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fbc11f7c9d2d2573ac0526b9da3e26a92cbdfdfb85b3a754b3c36be2fa3065a" Nov 03 04:35:53.921053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:53.919013 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj" Nov 03 04:35:55.016777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:55.016741 5275 scope.go:122] "RemoveContainer" containerID="7de017fb1698c774667a3e1ecbb9bef07c33807023993b3cadf021660e245e99" Nov 03 04:35:55.057806 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:35:55.057412 5275 scope.go:122] "RemoveContainer" containerID="5840d055660af59e1c56cc85286b6503abee2565b2c27532b4aad82c8b36d343" Nov 03 04:36:02.329174 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:36:02.329140 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:36:13.458990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:13.458956 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:36:18.479997 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:18.479954 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:36:18.480757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:18.480716 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:36:19.780134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.780100 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w"] Nov 03 04:36:19.780559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.780546 5275 cpu_manager.go:414] "containerMap: removing container" podUID="dda6461f-36bc-445a-bc34-07298002e42e" containerName="revoke-keys" Nov 03 04:36:19.780612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.780562 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda6461f-36bc-445a-bc34-07298002e42e" containerName="revoke-keys" Nov 03 04:36:19.780661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.780653 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda6461f-36bc-445a-bc34-07298002e42e" containerName="revoke-keys" Nov 03 04:36:19.783796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.783770 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.840831 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.840795 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:36:19.843265 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.843236 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kcw\" (UniqueName: \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.843631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.843278 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.843631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.843403 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.843909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.843884 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w"] Nov 03 04:36:19.890381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.890351 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:36:19.890566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.890430 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:36:19.890566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.890430 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:36:19.944982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.944943 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kcw\" (UniqueName: \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.945279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.945002 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.945279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.945050 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.945818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.945794 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:19.947608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:19.947585 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:20.006213 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:20.006177 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kcw\" (UniqueName: \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\") pod \"maas-api-revoke-sub-3e31a506-thn6w\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:20.096290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:20.096201 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:20.470711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:20.470687 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w"] Nov 03 04:36:20.473387 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:36:20.473358 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70afcc0_7844_4d66_86d3_f16145a105e0.slice/crio-dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570 WatchSource:0}: Error finding container dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570: Status 404 returned error can't find the container with id dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570 Nov 03 04:36:21.182075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:21.182029 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" event={"ID":"a70afcc0-7844-4d66-86d3-f16145a105e0","Type":"ContainerStarted","Data":"62bbbb6269f8d4891d3f8c19b36bd38d27172d6d5e41aada5ac09017aafdb572"} Nov 03 04:36:21.182541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:21.182095 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" event={"ID":"a70afcc0-7844-4d66-86d3-f16145a105e0","Type":"ContainerStarted","Data":"dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570"} Nov 03 04:36:21.298621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:21.298567 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" podStartSLOduration=2.298548876 podStartE2EDuration="2.298548876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:36:21.297076105 +0000 UTC m=+5672.449764527" watchObservedRunningTime="2026-11-03 04:36:21.298548876 +0000 UTC m=+5672.451237304" Nov 03 04:36:22.189445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:22.189402 5275 generic.go:356] "Generic (PLEG): container finished" podID="a70afcc0-7844-4d66-86d3-f16145a105e0" containerID="62bbbb6269f8d4891d3f8c19b36bd38d27172d6d5e41aada5ac09017aafdb572" exitCode=0 Nov 03 04:36:22.189876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:22.189452 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" event={"ID":"a70afcc0-7844-4d66-86d3-f16145a105e0","Type":"ContainerDied","Data":"62bbbb6269f8d4891d3f8c19b36bd38d27172d6d5e41aada5ac09017aafdb572"} Nov 03 04:36:23.347540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.347514 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:23.482177 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.482089 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\") pod \"a70afcc0-7844-4d66-86d3-f16145a105e0\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " Nov 03 04:36:23.482357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.482195 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\" (UniqueName: \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\") pod \"a70afcc0-7844-4d66-86d3-f16145a105e0\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " Nov 03 04:36:23.482357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.482249 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\") pod \"a70afcc0-7844-4d66-86d3-f16145a105e0\" (UID: \"a70afcc0-7844-4d66-86d3-f16145a105e0\") " Nov 03 04:36:23.482606 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.482573 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca" pod "a70afcc0-7844-4d66-86d3-f16145a105e0" (UID: "a70afcc0-7844-4d66-86d3-f16145a105e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:36:23.484944 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.484817 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw" pod "a70afcc0-7844-4d66-86d3-f16145a105e0" (UID: "a70afcc0-7844-4d66-86d3-f16145a105e0"). InnerVolumeSpecName "kube-api-access-69kcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:36:23.486661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.486636 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca" pod "a70afcc0-7844-4d66-86d3-f16145a105e0" (UID: "a70afcc0-7844-4d66-86d3-f16145a105e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:36:23.584019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.583981 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:23.584019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.584013 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kcw\" (UniqueName: \"kubernetes.io/projected/a70afcc0-7844-4d66-86d3-f16145a105e0-kube-api-access-69kcw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:23.584019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:23.584025 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70afcc0-7844-4d66-86d3-f16145a105e0-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:24.221221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:24.221167 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" event={"ID":"a70afcc0-7844-4d66-86d3-f16145a105e0","Type":"ContainerDied","Data":"dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570"} Nov 03 04:36:24.221445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:24.221274 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd85464a2e246af5c44a571ceae721d4a4246022ca5511153fe1a7a461ba570" Nov 03 04:36:24.221445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:24.221302 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w" Nov 03 04:36:28.479613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.479572 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerName="main" containerID="cri-o://ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453" gracePeriod=45 Nov 03 04:36:28.740295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.740227 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:36:28.890110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890065 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890105 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890155 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890200 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\" (UniqueName: \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890220 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890319 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\") pod \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\" (UID: \"36087bc1-d41f-432c-b9d5-36dd3bea7fb1\") " Nov 03 04:36:28.890628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890401 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:36:28.890628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890463 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:36:28.890628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890530 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:36:28.890803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890713 5275 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-model-cache\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:28.890803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890734 5275 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tmp-dir\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:28.890803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.890749 5275 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-home\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:28.892539 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.892509 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:36:28.892660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.892553 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "kube-api-access-htj22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:36:28.892660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.892599 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs" pod "36087bc1-d41f-432c-b9d5-36dd3bea7fb1" (UID: "36087bc1-d41f-432c-b9d5-36dd3bea7fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:36:28.991307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.991283 5275 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-dshm\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:28.991414 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.991312 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htj22\" (UniqueName: \"kubernetes.io/projected/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-kube-api-access-htj22\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:28.991414 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:28.991323 5275 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36087bc1-d41f-432c-b9d5-36dd3bea7fb1-tls-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:29.265900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.265792 5275 generic.go:356] "Generic (PLEG): container finished" podID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerID="ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453" exitCode=0 Nov 03 04:36:29.265900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.265832 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" Nov 03 04:36:29.265900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.265837 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" event={"ID":"36087bc1-d41f-432c-b9d5-36dd3bea7fb1","Type":"ContainerDied","Data":"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453"} Nov 03 04:36:29.266170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.265905 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" event={"ID":"36087bc1-d41f-432c-b9d5-36dd3bea7fb1","Type":"ContainerDied","Data":"154cfae17d92f9fcb1b38e414a40e5471684b1167e55a12f6075fe32ad2d04bb"} Nov 03 04:36:29.266170 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.265922 5275 scope.go:122] "RemoveContainer" containerID="ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453" Nov 03 04:36:29.275431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.275405 5275 scope.go:122] "RemoveContainer" containerID="ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453" Nov 03 04:36:29.275716 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:36:29.275691 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453\": container with ID starting with ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453 not found: ID does not exist" containerID="ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453" Nov 03 04:36:29.275773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.275725 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453"} err="failed to get container status \"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453\": rpc error: code = NotFound desc = could not find container \"ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453\": container with ID starting with ebac189d38f53b8935fc06fc813f4fc08e80cab095285e57676600ce01e0c453 not found: ID does not exist" Nov 03 04:36:29.429511 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.429475 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:36:29.478589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.478552 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6"] Nov 03 04:36:29.732763 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:29.732710 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-82587b78-kserve-5486cd4bd9-chfj6" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 04:36:30.984115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:30.984078 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" path="/var/lib/kubelet/pods/36087bc1-d41f-432c-b9d5-36dd3bea7fb1/volumes" Nov 03 04:36:32.328531 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:36:32.328494 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:36:45.230678 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.230641 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn"] Nov 03 04:36:45.231105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231064 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a70afcc0-7844-4d66-86d3-f16145a105e0" containerName="revoke-keys" Nov 03 04:36:45.231105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231076 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70afcc0-7844-4d66-86d3-f16145a105e0" containerName="revoke-keys" Nov 03 04:36:45.231105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231096 5275 cpu_manager.go:414] "containerMap: removing container" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerName="main" Nov 03 04:36:45.231105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231101 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerName="main" Nov 03 04:36:45.231242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231176 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70afcc0-7844-4d66-86d3-f16145a105e0" containerName="revoke-keys" Nov 03 04:36:45.231242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.231189 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="36087bc1-d41f-432c-b9d5-36dd3bea7fb1" containerName="main" Nov 03 04:36:45.234531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.234513 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.289421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.289383 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:36:45.295019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.294963 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.295019 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.295034 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.295285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.295190 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9xp\" (UniqueName: \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.338945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.338907 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:36:45.338945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.338933 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:36:45.338945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.338907 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:36:45.341114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.341094 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn"] Nov 03 04:36:45.396256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.396214 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.396256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.396263 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.396483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.396313 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9xp\" (UniqueName: \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.397105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.397073 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.398882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.398841 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.457399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.457361 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9xp\" (UniqueName: \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\") pod \"maas-api-revoke-sub-ce151bc8-w9tgn\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.547478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.547443 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:45.923021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:45.922992 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn"] Nov 03 04:36:45.925290 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:36:45.925245 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7658d1ea_5dc7_464a_9e17_9ea279cc4aca.slice/crio-79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273 WatchSource:0}: Error finding container 79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273: Status 404 returned error can't find the container with id 79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273 Nov 03 04:36:46.443878 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:46.443823 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" event={"ID":"7658d1ea-5dc7-464a-9e17-9ea279cc4aca","Type":"ContainerStarted","Data":"3231cd515adf2c29da3e8831fa7c342e4166b95983c732c5c6ea4018b4fa680d"} Nov 03 04:36:46.443878 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:46.443891 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" event={"ID":"7658d1ea-5dc7-464a-9e17-9ea279cc4aca","Type":"ContainerStarted","Data":"79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273"} Nov 03 04:36:46.562150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:46.562089 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" podStartSLOduration=1.562074515 podStartE2EDuration="1.562074515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:36:46.559453688 +0000 UTC m=+5697.712142108" watchObservedRunningTime="2026-11-03 04:36:46.562074515 +0000 UTC m=+5697.714762936" Nov 03 04:36:47.451443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:47.451406 5275 generic.go:356] "Generic (PLEG): container finished" podID="7658d1ea-5dc7-464a-9e17-9ea279cc4aca" containerID="3231cd515adf2c29da3e8831fa7c342e4166b95983c732c5c6ea4018b4fa680d" exitCode=0 Nov 03 04:36:47.451894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:47.451483 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" event={"ID":"7658d1ea-5dc7-464a-9e17-9ea279cc4aca","Type":"ContainerDied","Data":"3231cd515adf2c29da3e8831fa7c342e4166b95983c732c5c6ea4018b4fa680d"} Nov 03 04:36:48.598839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.598805 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:48.730281 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.730202 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\") pod \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " Nov 03 04:36:48.730281 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.730260 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\" (UniqueName: \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\") pod \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " Nov 03 04:36:48.730538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.730324 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\") pod \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\" (UID: \"7658d1ea-5dc7-464a-9e17-9ea279cc4aca\") " Nov 03 04:36:48.730778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.730749 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca" pod "7658d1ea-5dc7-464a-9e17-9ea279cc4aca" (UID: "7658d1ea-5dc7-464a-9e17-9ea279cc4aca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:36:48.732467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.732441 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca" pod "7658d1ea-5dc7-464a-9e17-9ea279cc4aca" (UID: "7658d1ea-5dc7-464a-9e17-9ea279cc4aca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:36:48.732598 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.732525 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp" pod "7658d1ea-5dc7-464a-9e17-9ea279cc4aca" (UID: "7658d1ea-5dc7-464a-9e17-9ea279cc4aca"). InnerVolumeSpecName "kube-api-access-qt9xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:36:48.831578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.831540 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:48.831578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.831571 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9xp\" (UniqueName: \"kubernetes.io/projected/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-kube-api-access-qt9xp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:48.831578 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:48.831581 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7658d1ea-5dc7-464a-9e17-9ea279cc4aca-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:36:49.464300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:49.464272 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" Nov 03 04:36:49.464575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:49.464306 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn" event={"ID":"7658d1ea-5dc7-464a-9e17-9ea279cc4aca","Type":"ContainerDied","Data":"79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273"} Nov 03 04:36:49.464575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:49.464352 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79be785e9524249efae1d684444c7940158c6aefd7d09d8d9521ea2bc64cf273" Nov 03 04:36:50.571088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:50.571057 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:36:50.613477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:50.613448 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:36:55.160216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:55.160178 5275 scope.go:122] "RemoveContainer" containerID="c193e9eef36112ccf89475ac0809bed9b9e2183a24fd3ddd307f123bc8e9a7e9" Nov 03 04:36:55.169571 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:36:55.169545 5275 scope.go:122] "RemoveContainer" containerID="78cd632c484ffef3636d7fefbab58b0bd5a01a9d6e1bb66ab8f444d97e1cdf39" Nov 03 04:37:02.202252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.202209 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:02.202894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.202875 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7658d1ea-5dc7-464a-9e17-9ea279cc4aca" containerName="revoke-keys" Nov 03 04:37:02.202949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.202899 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7658d1ea-5dc7-464a-9e17-9ea279cc4aca" containerName="revoke-keys" Nov 03 04:37:02.203008 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.202997 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7658d1ea-5dc7-464a-9e17-9ea279cc4aca" containerName="revoke-keys" Nov 03 04:37:02.206373 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.206351 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.266417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.266383 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:02.331911 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:37:02.331873 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:37:02.381666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.381625 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwmp\" (UniqueName: \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.381666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.381669 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.483109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.482998 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwmp\" (UniqueName: \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.483109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.483064 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.483979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.483952 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.544390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.544349 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwmp\" (UniqueName: \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\") pod \"payload-pre-processing-bd7f9cc7-w9bfc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:02.819899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:02.819833 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:03.193706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:03.193672 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:03.196711 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:37:03.196685 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a94c884_3193_46e1_8032_020f913a64bc.slice/crio-4ec7d18a7360e10fa9a6bb25fb01b154b3212a4bcaedc8b426e633aaaacdd581 WatchSource:0}: Error finding container 4ec7d18a7360e10fa9a6bb25fb01b154b3212a4bcaedc8b426e633aaaacdd581: Status 404 returned error can't find the container with id 4ec7d18a7360e10fa9a6bb25fb01b154b3212a4bcaedc8b426e633aaaacdd581 Nov 03 04:37:03.544632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:03.544593 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" event={"ID":"4a94c884-3193-46e1-8032-020f913a64bc","Type":"ContainerStarted","Data":"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29"} Nov 03 04:37:03.545088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:03.544653 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" event={"ID":"4a94c884-3193-46e1-8032-020f913a64bc","Type":"ContainerStarted","Data":"4ec7d18a7360e10fa9a6bb25fb01b154b3212a4bcaedc8b426e633aaaacdd581"} Nov 03 04:37:03.545088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:03.544782 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:03.660752 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:03.660699 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" podStartSLOduration=1.660684396 podStartE2EDuration="1.660684396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:37:03.659493929 +0000 UTC m=+5714.812182349" watchObservedRunningTime="2026-11-03 04:37:03.660684396 +0000 UTC m=+5714.813372818" Nov 03 04:37:14.552324 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:14.552295 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:16.883404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:16.883366 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:16.887190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:16.887167 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:16.928934 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:16.928901 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:16.929119 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:16.929089 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxgv\" (UniqueName: \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:16.945416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:16.945367 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:17.029817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.029761 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxgv\" (UniqueName: \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:17.030091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.030006 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:17.030983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.030943 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:17.095940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.095903 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxgv\" (UniqueName: \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\") pod \"payload-processing-76cbcc6f97-bnzzz\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:17.200794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.200696 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:17.576622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.576592 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:17.578719 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:37:17.578690 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075e6153_af1d_4b05_9db0_c0b19cb33cc0.slice/crio-ede58f5aa607ce7fb019226efa7ea648a8be5c5e8ea9298ecf1fcf8f1060eace WatchSource:0}: Error finding container ede58f5aa607ce7fb019226efa7ea648a8be5c5e8ea9298ecf1fcf8f1060eace: Status 404 returned error can't find the container with id ede58f5aa607ce7fb019226efa7ea648a8be5c5e8ea9298ecf1fcf8f1060eace Nov 03 04:37:17.623773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:17.623726 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" event={"ID":"075e6153-af1d-4b05-9db0-c0b19cb33cc0","Type":"ContainerStarted","Data":"ede58f5aa607ce7fb019226efa7ea648a8be5c5e8ea9298ecf1fcf8f1060eace"} Nov 03 04:37:18.630726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:18.630688 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" event={"ID":"075e6153-af1d-4b05-9db0-c0b19cb33cc0","Type":"ContainerStarted","Data":"9c0d3a448e1c8cf1f1634740d8dbcd2d40d049e24d1671daa6944e58fe55c40a"} Nov 03 04:37:18.631348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:18.630776 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:18.747590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:18.747528 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" podStartSLOduration=2.747509283 podStartE2EDuration="2.747509283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:37:18.746815734 +0000 UTC m=+5729.899504151" watchObservedRunningTime="2026-11-03 04:37:18.747509283 +0000 UTC m=+5729.900197705" Nov 03 04:37:29.641116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:29.641087 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:29.809335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:29.809303 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 04:37:32.330284 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:37:32.330245 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:37:34.809675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:34.809631 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" podUID="59ec2be7-fff9-4367-b67e-83e32e0f0586" containerName="payload-processing" containerID="cri-o://c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983" gracePeriod=25 Nov 03 04:37:35.059378 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.059351 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 04:37:35.217406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.217324 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\") pod \"59ec2be7-fff9-4367-b67e-83e32e0f0586\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " Nov 03 04:37:35.217612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.217517 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\" (UniqueName: \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\") pod \"59ec2be7-fff9-4367-b67e-83e32e0f0586\" (UID: \"59ec2be7-fff9-4367-b67e-83e32e0f0586\") " Nov 03 04:37:35.217716 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.217693 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume" pod "59ec2be7-fff9-4367-b67e-83e32e0f0586" (UID: "59ec2be7-fff9-4367-b67e-83e32e0f0586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:37:35.217898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.217834 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec2be7-fff9-4367-b67e-83e32e0f0586-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:35.219804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.219774 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j" pod "59ec2be7-fff9-4367-b67e-83e32e0f0586" (UID: "59ec2be7-fff9-4367-b67e-83e32e0f0586"). InnerVolumeSpecName "kube-api-access-dgf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:37:35.318828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.318787 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgf8j\" (UniqueName: \"kubernetes.io/projected/59ec2be7-fff9-4367-b67e-83e32e0f0586-kube-api-access-dgf8j\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:35.735658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.735617 5275 generic.go:356] "Generic (PLEG): container finished" podID="59ec2be7-fff9-4367-b67e-83e32e0f0586" containerID="c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983" exitCode=0 Nov 03 04:37:35.735886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.735690 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" Nov 03 04:37:35.735886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.735705 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" event={"ID":"59ec2be7-fff9-4367-b67e-83e32e0f0586","Type":"ContainerDied","Data":"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983"} Nov 03 04:37:35.735886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.735796 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-znlwv" event={"ID":"59ec2be7-fff9-4367-b67e-83e32e0f0586","Type":"ContainerDied","Data":"289cb0614abc436a42d67abc44b6a48d74f4cf1dacf2023e1cacd949a40c2508"} Nov 03 04:37:35.735886 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.735824 5275 scope.go:122] "RemoveContainer" containerID="c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983" Nov 03 04:37:35.745733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.745708 5275 scope.go:122] "RemoveContainer" containerID="c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983" Nov 03 04:37:35.746034 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:37:35.746013 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983\": container with ID starting with c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983 not found: ID does not exist" containerID="c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983" Nov 03 04:37:35.746106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.746043 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983"} err="failed to get container status \"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983\": rpc error: code = NotFound desc = could not find container \"c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983\": container with ID starting with c4b507dccba1d2fa210cb92364065c77d55c69a376641df25e82cea58aa79983 not found: ID does not exist" Nov 03 04:37:35.900713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.900671 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 04:37:35.950174 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:35.950141 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-znlwv"] Nov 03 04:37:36.982969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:36.982932 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ec2be7-fff9-4367-b67e-83e32e0f0586" path="/var/lib/kubelet/pods/59ec2be7-fff9-4367-b67e-83e32e0f0586/volumes" Nov 03 04:37:44.581733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:44.581694 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:44.699820 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:44.699723 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:49.582724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.582649 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerName="payload-processing" containerID="cri-o://9c0d3a448e1c8cf1f1634740d8dbcd2d40d049e24d1671daa6944e58fe55c40a" gracePeriod=25 Nov 03 04:37:49.638861 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:37:49.638825 5275 logging.go:55] [core] [Channel #16008 SubChannel #16009]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.98:50052", ServerName: "10.133.1.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.98:50052: connect: connection refused" Nov 03 04:37:49.700491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.700412 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" podUID="4a94c884-3193-46e1-8032-020f913a64bc" containerName="payload-pre-processing" containerID="cri-o://dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29" gracePeriod=25 Nov 03 04:37:49.813734 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.813692 5275 generic.go:356] "Generic (PLEG): container finished" podID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerID="9c0d3a448e1c8cf1f1634740d8dbcd2d40d049e24d1671daa6944e58fe55c40a" exitCode=0 Nov 03 04:37:49.813926 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.813756 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" event={"ID":"075e6153-af1d-4b05-9db0-c0b19cb33cc0","Type":"ContainerDied","Data":"9c0d3a448e1c8cf1f1634740d8dbcd2d40d049e24d1671daa6944e58fe55c40a"} Nov 03 04:37:49.846740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.846663 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:49.956429 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.956400 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:49.962732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.962703 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\" (UniqueName: \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\") pod \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " Nov 03 04:37:49.963896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.963594 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\") pod \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\" (UID: \"075e6153-af1d-4b05-9db0-c0b19cb33cc0\") " Nov 03 04:37:49.964207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.964179 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume" pod "075e6153-af1d-4b05-9db0-c0b19cb33cc0" (UID: "075e6153-af1d-4b05-9db0-c0b19cb33cc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:37:49.965729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:49.965707 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv" pod "075e6153-af1d-4b05-9db0-c0b19cb33cc0" (UID: "075e6153-af1d-4b05-9db0-c0b19cb33cc0"). InnerVolumeSpecName "kube-api-access-cvxgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:37:50.064983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.064943 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\") pod \"4a94c884-3193-46e1-8032-020f913a64bc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " Nov 03 04:37:50.064983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.064992 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\" (UniqueName: \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\") pod \"4a94c884-3193-46e1-8032-020f913a64bc\" (UID: \"4a94c884-3193-46e1-8032-020f913a64bc\") " Nov 03 04:37:50.065351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.065329 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume" pod "4a94c884-3193-46e1-8032-020f913a64bc" (UID: "4a94c884-3193-46e1-8032-020f913a64bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:37:50.065467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.065448 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a94c884-3193-46e1-8032-020f913a64bc-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:50.065467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.065467 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxgv\" (UniqueName: \"kubernetes.io/projected/075e6153-af1d-4b05-9db0-c0b19cb33cc0-kube-api-access-cvxgv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:50.065596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.065477 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075e6153-af1d-4b05-9db0-c0b19cb33cc0-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:50.067388 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.067367 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp" pod "4a94c884-3193-46e1-8032-020f913a64bc" (UID: "4a94c884-3193-46e1-8032-020f913a64bc"). InnerVolumeSpecName "kube-api-access-qvwmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:37:50.166798 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.166722 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwmp\" (UniqueName: \"kubernetes.io/projected/4a94c884-3193-46e1-8032-020f913a64bc-kube-api-access-qvwmp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:37:50.639453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.639406 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.98:50052\" within 1s: context deadline exceeded" Nov 03 04:37:50.639897 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:37:50.639482 5275 logging.go:55] [core] [Channel #16008 SubChannel #16009]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.98:50052", ServerName: "10.133.1.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.98:50052: operation was canceled" Nov 03 04:37:50.821767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.821738 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" Nov 03 04:37:50.821988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.821767 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76cbcc6f97-bnzzz" event={"ID":"075e6153-af1d-4b05-9db0-c0b19cb33cc0","Type":"ContainerDied","Data":"ede58f5aa607ce7fb019226efa7ea648a8be5c5e8ea9298ecf1fcf8f1060eace"} Nov 03 04:37:50.821988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.821840 5275 scope.go:122] "RemoveContainer" containerID="9c0d3a448e1c8cf1f1634740d8dbcd2d40d049e24d1671daa6944e58fe55c40a" Nov 03 04:37:50.823467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.823401 5275 generic.go:356] "Generic (PLEG): container finished" podID="4a94c884-3193-46e1-8032-020f913a64bc" containerID="dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29" exitCode=0 Nov 03 04:37:50.823467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.823451 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" event={"ID":"4a94c884-3193-46e1-8032-020f913a64bc","Type":"ContainerDied","Data":"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29"} Nov 03 04:37:50.823467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.823462 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" Nov 03 04:37:50.823676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.823491 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc" event={"ID":"4a94c884-3193-46e1-8032-020f913a64bc","Type":"ContainerDied","Data":"4ec7d18a7360e10fa9a6bb25fb01b154b3212a4bcaedc8b426e633aaaacdd581"} Nov 03 04:37:50.832394 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.832370 5275 scope.go:122] "RemoveContainer" containerID="dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29" Nov 03 04:37:50.841530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.841507 5275 scope.go:122] "RemoveContainer" containerID="dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29" Nov 03 04:37:50.841857 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:37:50.841821 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29\": container with ID starting with dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29 not found: ID does not exist" containerID="dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29" Nov 03 04:37:50.841924 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.841877 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29"} err="failed to get container status \"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29\": rpc error: code = NotFound desc = could not find container \"dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29\": container with ID starting with dfa7dfd3e8370bac011f1a01a95be6c755de07ecd5eae634ea57d968743afd29 not found: ID does not exist" Nov 03 04:37:50.989079 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:50.988995 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:51.038499 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.038455 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-76cbcc6f97-bnzzz"] Nov 03 04:37:51.149625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.149585 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:51.153996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.153965 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-bd7f9cc7-w9bfc"] Nov 03 04:37:51.715435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715395 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715816 5275 cpu_manager.go:414] "containerMap: removing container" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerName="payload-processing" Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715828 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerName="payload-processing" Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715862 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4a94c884-3193-46e1-8032-020f913a64bc" containerName="payload-pre-processing" Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715868 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a94c884-3193-46e1-8032-020f913a64bc" containerName="payload-pre-processing" Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715880 5275 cpu_manager.go:414] "containerMap: removing container" podUID="59ec2be7-fff9-4367-b67e-83e32e0f0586" containerName="payload-processing" Nov 03 04:37:51.715912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715885 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ec2be7-fff9-4367-b67e-83e32e0f0586" containerName="payload-processing" Nov 03 04:37:51.716202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715961 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" containerName="payload-processing" Nov 03 04:37:51.716202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715971 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a94c884-3193-46e1-8032-020f913a64bc" containerName="payload-pre-processing" Nov 03 04:37:51.716202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.715978 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ec2be7-fff9-4367-b67e-83e32e0f0586" containerName="payload-processing" Nov 03 04:37:51.720483 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.720465 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.776693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.775875 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 04:37:51.776693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.775927 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 04:37:51.776693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.775990 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 04:37:51.776693 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.776026 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 04:37:51.779379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.779346 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:37:51.782680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.782654 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.782837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.782703 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.782837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.782748 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.783009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.782882 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfhc\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.825582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.825553 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 04:37:51.825582 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.825554 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-sjmgs\"" Nov 03 04:37:51.883562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.883519 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfhc\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.883762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.883635 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.883762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.883677 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.883762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.883721 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.884476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.884443 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.884877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.884830 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.886650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.886623 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:51.944938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:51.944906 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfhc\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\") pod \"payload-processing-external-model-5bcd7997-nksq2\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:52.034535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.034498 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:52.422660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.422631 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:37:52.424485 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:37:52.424452 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7087daf4_defd_4b2c_8859_97dbf405ea02.slice/crio-80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6 WatchSource:0}: Error finding container 80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6: Status 404 returned error can't find the container with id 80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6 Nov 03 04:37:52.838307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.838260 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" event={"ID":"7087daf4-defd-4b2c-8859-97dbf405ea02","Type":"ContainerStarted","Data":"3198afe64a41b34dd8a886cfc4e734d700481895c2b871b6119b67f675e3614c"} Nov 03 04:37:52.838762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.838360 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" event={"ID":"7087daf4-defd-4b2c-8859-97dbf405ea02","Type":"ContainerStarted","Data":"80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6"} Nov 03 04:37:52.838762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.838523 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:37:52.956072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.956014 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" podStartSLOduration=1.955998157 podStartE2EDuration="1.955998157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:37:52.953873164 +0000 UTC m=+5764.106561585" watchObservedRunningTime="2026-11-03 04:37:52.955998157 +0000 UTC m=+5764.108686582" Nov 03 04:37:52.985118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.985080 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075e6153-af1d-4b05-9db0-c0b19cb33cc0" path="/var/lib/kubelet/pods/075e6153-af1d-4b05-9db0-c0b19cb33cc0/volumes" Nov 03 04:37:52.985641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:37:52.985624 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a94c884-3193-46e1-8032-020f913a64bc" path="/var/lib/kubelet/pods/4a94c884-3193-46e1-8032-020f913a64bc/volumes" Nov 03 04:38:02.330029 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:38:02.329994 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:38:03.855819 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:38:03.855788 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:38:23.178370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:38:23.178333 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7"] Nov 03 04:38:23.227432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:38:23.227397 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4b3372-vttp7"] Nov 03 04:38:24.982863 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:38:24.982815 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd298e5-6957-4db1-b4e6-58b15ff7712d" path="/var/lib/kubelet/pods/1bd298e5-6957-4db1-b4e6-58b15ff7712d/volumes" Nov 03 04:38:32.328793 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:38:32.328756 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:39:02.329560 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:39:02.329514 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:39:16.168076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.168036 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm"] Nov 03 04:39:16.171656 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.171639 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.224189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.224157 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:39:16.273929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.273899 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:39:16.276264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.276238 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm"] Nov 03 04:39:16.276550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.276534 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:39:16.305368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.305332 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.305528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.305375 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2z8\" (UniqueName: \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.305576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.305547 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.324624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.324588 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:39:16.406695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.406656 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.406923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.406725 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.406923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.406752 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2z8\" (UniqueName: \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.407460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.407430 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.409338 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.409311 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.468672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.468599 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2z8\" (UniqueName: \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\") pod \"maas-api-revoke-sub-e2547622-jl2tm\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.485671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.485636 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:16.861286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:16.861253 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm"] Nov 03 04:39:16.863772 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:39:16.863733 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad4f8cc_8ef5_4af7_afc6_b967ee4419e0.slice/crio-17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299 WatchSource:0}: Error finding container 17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299: Status 404 returned error can't find the container with id 17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299 Nov 03 04:39:17.359435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:17.359389 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" event={"ID":"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0","Type":"ContainerStarted","Data":"a76a45064e88e1704ef0e0a0a6412f6cf2f210fa927475ca1695f4e4439026a7"} Nov 03 04:39:17.359862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:17.359463 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" event={"ID":"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0","Type":"ContainerStarted","Data":"17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299"} Nov 03 04:39:17.477445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:17.477339 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" podStartSLOduration=1.47732402 podStartE2EDuration="1.47732402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:39:17.474696592 +0000 UTC m=+5848.627385014" watchObservedRunningTime="2026-11-03 04:39:17.47732402 +0000 UTC m=+5848.630012441" Nov 03 04:39:18.365726 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:18.365688 5275 generic.go:356] "Generic (PLEG): container finished" podID="3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" containerID="a76a45064e88e1704ef0e0a0a6412f6cf2f210fa927475ca1695f4e4439026a7" exitCode=0 Nov 03 04:39:18.366180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:18.365776 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" event={"ID":"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0","Type":"ContainerDied","Data":"a76a45064e88e1704ef0e0a0a6412f6cf2f210fa927475ca1695f4e4439026a7"} Nov 03 04:39:19.517176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.517148 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:19.638943 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.638826 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\") pod \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " Nov 03 04:39:19.639132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.638948 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\") pod \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " Nov 03 04:39:19.639132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.639124 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\" (UniqueName: \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\") pod \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\" (UID: \"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0\") " Nov 03 04:39:19.639306 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.639282 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca" pod "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" (UID: "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:39:19.639583 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.639544 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:19.641711 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.641680 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca" pod "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" (UID: "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:39:19.643028 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.642986 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8" pod "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" (UID: "3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0"). InnerVolumeSpecName "kube-api-access-gr2z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:39:19.740208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.740167 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2z8\" (UniqueName: \"kubernetes.io/projected/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-kube-api-access-gr2z8\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:19.740208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:19.740203 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:20.378075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:20.378040 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" event={"ID":"3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0","Type":"ContainerDied","Data":"17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299"} Nov 03 04:39:20.378075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:20.378074 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm" Nov 03 04:39:20.378333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:20.378095 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17cadd38989d8b9d77366b15f0d68d75837ce528f326a1cf2ee3531844796299" Nov 03 04:39:27.742512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:27.742432 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:39:32.460085 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:39:32.460042 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:39:32.744101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:32.743938 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" podUID="7087daf4-defd-4b2c-8859-97dbf405ea02" containerName="payload-processing" containerID="cri-o://3198afe64a41b34dd8a886cfc4e734d700481895c2b871b6119b67f675e3614c" gracePeriod=25 Nov 03 04:39:33.484984 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.484943 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:39:33.485610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.485570 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:39:33.519232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.517007 5275 generic.go:356] "Generic (PLEG): container finished" podID="7087daf4-defd-4b2c-8859-97dbf405ea02" containerID="3198afe64a41b34dd8a886cfc4e734d700481895c2b871b6119b67f675e3614c" exitCode=0 Nov 03 04:39:33.519232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.517200 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" event={"ID":"7087daf4-defd-4b2c-8859-97dbf405ea02","Type":"ContainerDied","Data":"3198afe64a41b34dd8a886cfc4e734d700481895c2b871b6119b67f675e3614c"} Nov 03 04:39:33.519232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.517253 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" event={"ID":"7087daf4-defd-4b2c-8859-97dbf405ea02","Type":"ContainerDied","Data":"80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6"} Nov 03 04:39:33.519232 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.517278 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80017f5dc93899347b5cccae1547fa6265cc202080685351868c9e82224012c6" Nov 03 04:39:33.535207 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.534603 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:39:33.651759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.651711 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\") pod \"7087daf4-defd-4b2c-8859-97dbf405ea02\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " Nov 03 04:39:33.657570 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.657535 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\") pod \"7087daf4-defd-4b2c-8859-97dbf405ea02\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " Nov 03 04:39:33.657879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.657829 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\") pod \"7087daf4-defd-4b2c-8859-97dbf405ea02\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " Nov 03 04:39:33.658581 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.658559 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\") pod \"7087daf4-defd-4b2c-8859-97dbf405ea02\" (UID: \"7087daf4-defd-4b2c-8859-97dbf405ea02\") " Nov 03 04:39:33.659565 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.659538 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume" pod "7087daf4-defd-4b2c-8859-97dbf405ea02" (UID: "7087daf4-defd-4b2c-8859-97dbf405ea02"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:39:33.659962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.659939 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume" pod "7087daf4-defd-4b2c-8859-97dbf405ea02" (UID: "7087daf4-defd-4b2c-8859-97dbf405ea02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:39:33.662741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.662719 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials" pod "7087daf4-defd-4b2c-8859-97dbf405ea02" (UID: "7087daf4-defd-4b2c-8859-97dbf405ea02"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:39:33.667273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.667246 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc" pod "7087daf4-defd-4b2c-8859-97dbf405ea02" (UID: "7087daf4-defd-4b2c-8859-97dbf405ea02"). InnerVolumeSpecName "kube-api-access-tcfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:39:33.760551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.760508 5275 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-routing-overlay-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:33.760551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.760544 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcfhc\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-kube-api-access-tcfhc\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:33.760551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.760553 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7087daf4-defd-4b2c-8859-97dbf405ea02-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:33.760551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:33.760562 5275 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7087daf4-defd-4b2c-8859-97dbf405ea02-provider-credentials\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:34.523041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:34.523005 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nksq2" Nov 03 04:39:34.687224 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:34.687186 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:39:34.736688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:34.736646 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nksq2"] Nov 03 04:39:34.983075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:34.982989 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7087daf4-defd-4b2c-8859-97dbf405ea02" path="/var/lib/kubelet/pods/7087daf4-defd-4b2c-8859-97dbf405ea02/volumes" Nov 03 04:39:48.309889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.309824 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f"] Nov 03 04:39:48.310541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310516 5275 cpu_manager.go:414] "containerMap: removing container" podUID="7087daf4-defd-4b2c-8859-97dbf405ea02" containerName="payload-processing" Nov 03 04:39:48.310632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310544 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7087daf4-defd-4b2c-8859-97dbf405ea02" containerName="payload-processing" Nov 03 04:39:48.310632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310570 5275 cpu_manager.go:414] "containerMap: removing container" podUID="3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" containerName="revoke-keys" Nov 03 04:39:48.310632 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310579 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" containerName="revoke-keys" Nov 03 04:39:48.310803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310691 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="7087daf4-defd-4b2c-8859-97dbf405ea02" containerName="payload-processing" Nov 03 04:39:48.310803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.310704 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" containerName="revoke-keys" Nov 03 04:39:48.315488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.315460 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.367793 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.367741 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:39:48.383989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.383943 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.384198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.384034 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpvq\" (UniqueName: \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.384198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.384112 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.417250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.417202 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:39:48.417250 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.417247 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:39:48.417524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.417245 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:39:48.420066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.420034 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f"] Nov 03 04:39:48.484911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.484871 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.485134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.484938 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpvq\" (UniqueName: \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.485134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.484984 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.485764 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.485683 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.487516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.487496 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.546868 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.546819 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpvq\" (UniqueName: \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\") pod \"maas-api-revoke-sub-c1e271a9-bg94f\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:48.630976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:48.630887 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:49.008618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:49.008595 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f"] Nov 03 04:39:49.010264 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:39:49.010238 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bbd37e_1bbc_4ac9_9e06_946f16285046.slice/crio-5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754 WatchSource:0}: Error finding container 5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754: Status 404 returned error can't find the container with id 5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754 Nov 03 04:39:49.665807 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:49.665763 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" event={"ID":"73bbd37e-1bbc-4ac9-9e06-946f16285046","Type":"ContainerStarted","Data":"23c84c841efbeb286187d278223f1cf56475c20190858e6f50f13bd2392eb7cd"} Nov 03 04:39:49.666273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:49.665885 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" event={"ID":"73bbd37e-1bbc-4ac9-9e06-946f16285046","Type":"ContainerStarted","Data":"5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754"} Nov 03 04:39:49.832038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:49.831975 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" podStartSLOduration=1.831960995 podStartE2EDuration="1.831960995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:39:49.78134994 +0000 UTC m=+5880.934038361" watchObservedRunningTime="2026-11-03 04:39:49.831960995 +0000 UTC m=+5880.984649450" Nov 03 04:39:50.673818 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:50.673786 5275 generic.go:356] "Generic (PLEG): container finished" podID="73bbd37e-1bbc-4ac9-9e06-946f16285046" containerID="23c84c841efbeb286187d278223f1cf56475c20190858e6f50f13bd2392eb7cd" exitCode=0 Nov 03 04:39:50.674252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:50.673829 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" event={"ID":"73bbd37e-1bbc-4ac9-9e06-946f16285046","Type":"ContainerDied","Data":"23c84c841efbeb286187d278223f1cf56475c20190858e6f50f13bd2392eb7cd"} Nov 03 04:39:51.815485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.815457 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:51.911389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.911356 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\") pod \"73bbd37e-1bbc-4ac9-9e06-946f16285046\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " Nov 03 04:39:51.911598 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.911403 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\") pod \"73bbd37e-1bbc-4ac9-9e06-946f16285046\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " Nov 03 04:39:51.911598 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.911430 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\" (UniqueName: \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\") pod \"73bbd37e-1bbc-4ac9-9e06-946f16285046\" (UID: \"73bbd37e-1bbc-4ac9-9e06-946f16285046\") " Nov 03 04:39:51.911737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.911700 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca" pod "73bbd37e-1bbc-4ac9-9e06-946f16285046" (UID: "73bbd37e-1bbc-4ac9-9e06-946f16285046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:39:51.913871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.913828 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca" pod "73bbd37e-1bbc-4ac9-9e06-946f16285046" (UID: "73bbd37e-1bbc-4ac9-9e06-946f16285046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:39:51.913999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:51.913866 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq" pod "73bbd37e-1bbc-4ac9-9e06-946f16285046" (UID: "73bbd37e-1bbc-4ac9-9e06-946f16285046"). InnerVolumeSpecName "kube-api-access-7lpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:39:52.012810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.012725 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:52.012810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.012756 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bbd37e-1bbc-4ac9-9e06-946f16285046-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:52.012810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.012766 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpvq\" (UniqueName: \"kubernetes.io/projected/73bbd37e-1bbc-4ac9-9e06-946f16285046-kube-api-access-7lpvq\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:39:52.686229 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.686185 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" event={"ID":"73bbd37e-1bbc-4ac9-9e06-946f16285046","Type":"ContainerDied","Data":"5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754"} Nov 03 04:39:52.686417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.686260 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b02ca395b24ed23a9593542f8f60acd6deab9abf3938e2af4fc1002bacde754" Nov 03 04:39:52.686417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:52.686197 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f" Nov 03 04:39:55.349621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:39:55.349581 5275 scope.go:122] "RemoveContainer" containerID="16a2af183358e4e3c327b7514484f34b1dd1411def46c8c4d26b8ae16f6b8c09" Nov 03 04:40:02.330390 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:40:02.330349 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:40:14.390091 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.390049 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv"] Nov 03 04:40:14.390631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.390463 5275 cpu_manager.go:414] "containerMap: removing container" podUID="73bbd37e-1bbc-4ac9-9e06-946f16285046" containerName="revoke-keys" Nov 03 04:40:14.390631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.390476 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bbd37e-1bbc-4ac9-9e06-946f16285046" containerName="revoke-keys" Nov 03 04:40:14.390631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.390561 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bbd37e-1bbc-4ac9-9e06-946f16285046" containerName="revoke-keys" Nov 03 04:40:14.393661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.393637 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.495795 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.495745 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqkr\" (UniqueName: \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.496016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.495901 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.496016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.495970 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.498046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.498030 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:40:14.498117 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.498107 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:40:14.498158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.498143 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:40:14.498205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.498191 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:40:14.500395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.500372 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv"] Nov 03 04:40:14.597312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.597278 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.597515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.597336 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.597515 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.597390 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqkr\" (UniqueName: \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.598000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.597973 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.599826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.599806 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.658876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.658777 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqkr\" (UniqueName: \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\") pod \"maas-api-revoke-sub-2e1f5b5f-gnmqv\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:14.709555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:14.709518 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:15.084941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:15.084915 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv"] Nov 03 04:40:15.087003 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:40:15.086973 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4848c4f_e8a0_4813_8854_aa59cbf524d9.slice/crio-5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff WatchSource:0}: Error finding container 5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff: Status 404 returned error can't find the container with id 5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff Nov 03 04:40:15.843180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:15.843139 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" event={"ID":"c4848c4f-e8a0-4813-8854-aa59cbf524d9","Type":"ContainerStarted","Data":"e690ac7fdd20c04bf50536bdc9b8841a5bd30f1f6abf758c694d3f583c14113c"} Nov 03 04:40:15.843587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:15.843202 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" event={"ID":"c4848c4f-e8a0-4813-8854-aa59cbf524d9","Type":"ContainerStarted","Data":"5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff"} Nov 03 04:40:15.959299 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:15.959155 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" podStartSLOduration=1.9591343380000001 podStartE2EDuration="1.959134338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:40:15.958694903 +0000 UTC m=+5907.111383325" watchObservedRunningTime="2026-11-03 04:40:15.959134338 +0000 UTC m=+5907.111822765" Nov 03 04:40:16.850342 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:16.850293 5275 generic.go:356] "Generic (PLEG): container finished" podID="c4848c4f-e8a0-4813-8854-aa59cbf524d9" containerID="e690ac7fdd20c04bf50536bdc9b8841a5bd30f1f6abf758c694d3f583c14113c" exitCode=0 Nov 03 04:40:16.850884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:16.850387 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" event={"ID":"c4848c4f-e8a0-4813-8854-aa59cbf524d9","Type":"ContainerDied","Data":"e690ac7fdd20c04bf50536bdc9b8841a5bd30f1f6abf758c694d3f583c14113c"} Nov 03 04:40:18.001366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.001339 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:18.136681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.136595 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\") pod \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " Nov 03 04:40:18.136681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.136650 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\") pod \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " Nov 03 04:40:18.136920 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.136686 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\" (UniqueName: \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\") pod \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\" (UID: \"c4848c4f-e8a0-4813-8854-aa59cbf524d9\") " Nov 03 04:40:18.137146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.137113 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca" pod "c4848c4f-e8a0-4813-8854-aa59cbf524d9" (UID: "c4848c4f-e8a0-4813-8854-aa59cbf524d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:40:18.139262 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.139231 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr" pod "c4848c4f-e8a0-4813-8854-aa59cbf524d9" (UID: "c4848c4f-e8a0-4813-8854-aa59cbf524d9"). InnerVolumeSpecName "kube-api-access-ldqkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:40:18.139426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.139400 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca" pod "c4848c4f-e8a0-4813-8854-aa59cbf524d9" (UID: "c4848c4f-e8a0-4813-8854-aa59cbf524d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:40:18.238009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.237959 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:18.238009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.238002 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4848c4f-e8a0-4813-8854-aa59cbf524d9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:18.238009 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.238016 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqkr\" (UniqueName: \"kubernetes.io/projected/c4848c4f-e8a0-4813-8854-aa59cbf524d9-kube-api-access-ldqkr\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:18.863478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.863440 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" event={"ID":"c4848c4f-e8a0-4813-8854-aa59cbf524d9","Type":"ContainerDied","Data":"5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff"} Nov 03 04:40:18.863685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.863540 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad4d1b182d0e9ce89982f341f472e356ebdbf1cfd6d4b3854c4fc923ba4f6ff" Nov 03 04:40:18.863685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:18.863452 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv" Nov 03 04:40:26.851732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.851695 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj"] Nov 03 04:40:26.852381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.852323 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c4848c4f-e8a0-4813-8854-aa59cbf524d9" containerName="revoke-keys" Nov 03 04:40:26.852381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.852346 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4848c4f-e8a0-4813-8854-aa59cbf524d9" containerName="revoke-keys" Nov 03 04:40:26.852519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.852488 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4848c4f-e8a0-4813-8854-aa59cbf524d9" containerName="revoke-keys" Nov 03 04:40:26.855982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.855959 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:26.910454 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.910418 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:40:26.925118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.925081 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:26.925337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.925310 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4w7\" (UniqueName: \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:26.925410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.925362 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:26.959584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.959546 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:40:26.959584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.959570 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:40:26.959829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.959595 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:40:26.962070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:26.962051 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj"] Nov 03 04:40:27.027021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.026982 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.027233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.027045 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.027233 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.027161 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4w7\" (UniqueName: \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.027949 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.027878 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.029627 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.029606 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.088561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.088521 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4w7\" (UniqueName: \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\") pod \"maas-api-revoke-sub-ec843018-s8wvj\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.169243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.169158 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:27.544216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.544185 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj"] Nov 03 04:40:27.545646 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:40:27.545612 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e812cad_c99d_4b40_9c2c_84da1580dbb5.slice/crio-88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f WatchSource:0}: Error finding container 88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f: Status 404 returned error can't find the container with id 88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f Nov 03 04:40:27.915782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:27.915736 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" event={"ID":"4e812cad-c99d-4b40-9c2c-84da1580dbb5","Type":"ContainerStarted","Data":"88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f"} Nov 03 04:40:28.922926 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:28.922814 5275 generic.go:356] "Generic (PLEG): container finished" podID="4e812cad-c99d-4b40-9c2c-84da1580dbb5" containerID="569402e2c73f46adb599ebcc554b8004b1ba9c8aa3d8bb01400fdca982aa4912" exitCode=0 Nov 03 04:40:28.922926 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:28.922872 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" event={"ID":"4e812cad-c99d-4b40-9c2c-84da1580dbb5","Type":"ContainerDied","Data":"569402e2c73f46adb599ebcc554b8004b1ba9c8aa3d8bb01400fdca982aa4912"} Nov 03 04:40:30.070898 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.070871 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:30.154140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.154103 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\") pod \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " Nov 03 04:40:30.154140 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.154147 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\") pod \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " Nov 03 04:40:30.154411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.154200 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\" (UniqueName: \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\") pod \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\" (UID: \"4e812cad-c99d-4b40-9c2c-84da1580dbb5\") " Nov 03 04:40:30.154597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.154574 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca" pod "4e812cad-c99d-4b40-9c2c-84da1580dbb5" (UID: "4e812cad-c99d-4b40-9c2c-84da1580dbb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:40:30.156442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.156415 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca" pod "4e812cad-c99d-4b40-9c2c-84da1580dbb5" (UID: "4e812cad-c99d-4b40-9c2c-84da1580dbb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:40:30.156601 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.156452 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7" pod "4e812cad-c99d-4b40-9c2c-84da1580dbb5" (UID: "4e812cad-c99d-4b40-9c2c-84da1580dbb5"). InnerVolumeSpecName "kube-api-access-wh4w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:40:30.255507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.255406 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:30.255507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.255444 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e812cad-c99d-4b40-9c2c-84da1580dbb5-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:30.255507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.255459 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4w7\" (UniqueName: \"kubernetes.io/projected/4e812cad-c99d-4b40-9c2c-84da1580dbb5-kube-api-access-wh4w7\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:40:30.936004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.935967 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" event={"ID":"4e812cad-c99d-4b40-9c2c-84da1580dbb5","Type":"ContainerDied","Data":"88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f"} Nov 03 04:40:30.936205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.936045 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88106c256eacdd9dac204b04e0bb83531f7a5dbdac9d6ef30221d5d5649a607f" Nov 03 04:40:30.936205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:30.935984 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj" Nov 03 04:40:32.329836 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:40:32.329798 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:40:54.112162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:54.112084 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj"] Nov 03 04:40:54.161744 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:54.161713 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t7qkj"] Nov 03 04:40:54.984478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:54.984436 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda6461f-36bc-445a-bc34-07298002e42e" path="/var/lib/kubelet/pods/dda6461f-36bc-445a-bc34-07298002e42e/volumes" Nov 03 04:40:55.371950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:40:55.371917 5275 scope.go:122] "RemoveContainer" containerID="d18bd19980cc83c4e52f91a9d846ac8040f894d2b839c73d269a8229b780b435" Nov 03 04:41:02.329898 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:41:02.329829 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:41:02.634144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.634055 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j"] Nov 03 04:41:02.634523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.634509 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4e812cad-c99d-4b40-9c2c-84da1580dbb5" containerName="revoke-keys" Nov 03 04:41:02.634566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.634525 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e812cad-c99d-4b40-9c2c-84da1580dbb5" containerName="revoke-keys" Nov 03 04:41:02.634617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.634606 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e812cad-c99d-4b40-9c2c-84da1580dbb5" containerName="revoke-keys" Nov 03 04:41:02.637695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.637678 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.657270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.657237 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.657422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.657299 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qc2l\" (UniqueName: \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.657477 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.657445 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.690269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.690231 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:41:02.690895 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.690873 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:41:02.740300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.740267 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:41:02.742649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.742627 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j"] Nov 03 04:41:02.757964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.757931 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.758126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.757994 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qc2l\" (UniqueName: \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.758126 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.758055 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.760624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.760597 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.793762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.793723 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:41:02.799201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.799176 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.821056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.821022 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qc2l\" (UniqueName: \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\") pod \"maas-api-revoke-sub-046e1343-vwp4j\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:02.950450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:02.950357 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:03.330702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:03.330679 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j"] Nov 03 04:41:03.333055 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:41:03.333025 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42768e93_b2a3_4a67_b2fc_c251755ee766.slice/crio-6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc WatchSource:0}: Error finding container 6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc: Status 404 returned error can't find the container with id 6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc Nov 03 04:41:04.175816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:04.175777 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" event={"ID":"42768e93-b2a3-4a67-b2fc-c251755ee766","Type":"ContainerStarted","Data":"b9c6745f1cb2401b5004fa5f3b90321df2d4cffdca26773994c558fe3d92d092"} Nov 03 04:41:04.176039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:04.175836 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" event={"ID":"42768e93-b2a3-4a67-b2fc-c251755ee766","Type":"ContainerStarted","Data":"6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc"} Nov 03 04:41:04.293096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:04.293041 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" podStartSLOduration=2.293024723 podStartE2EDuration="2.293024723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:41:04.291077421 +0000 UTC m=+5955.443765844" watchObservedRunningTime="2026-11-03 04:41:04.293024723 +0000 UTC m=+5955.445713144" Nov 03 04:41:05.183431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:05.183396 5275 generic.go:356] "Generic (PLEG): container finished" podID="42768e93-b2a3-4a67-b2fc-c251755ee766" containerID="b9c6745f1cb2401b5004fa5f3b90321df2d4cffdca26773994c558fe3d92d092" exitCode=0 Nov 03 04:41:05.183838 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:05.183459 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" event={"ID":"42768e93-b2a3-4a67-b2fc-c251755ee766","Type":"ContainerDied","Data":"b9c6745f1cb2401b5004fa5f3b90321df2d4cffdca26773994c558fe3d92d092"} Nov 03 04:41:06.326110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.326081 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:06.395650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.395609 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\" (UniqueName: \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\") pod \"42768e93-b2a3-4a67-b2fc-c251755ee766\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " Nov 03 04:41:06.395877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.395676 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\") pod \"42768e93-b2a3-4a67-b2fc-c251755ee766\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " Nov 03 04:41:06.395877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.395702 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\") pod \"42768e93-b2a3-4a67-b2fc-c251755ee766\" (UID: \"42768e93-b2a3-4a67-b2fc-c251755ee766\") " Nov 03 04:41:06.396150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.396125 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca" pod "42768e93-b2a3-4a67-b2fc-c251755ee766" (UID: "42768e93-b2a3-4a67-b2fc-c251755ee766"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:41:06.397930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.397903 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca" pod "42768e93-b2a3-4a67-b2fc-c251755ee766" (UID: "42768e93-b2a3-4a67-b2fc-c251755ee766"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:41:06.398004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.397930 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l" pod "42768e93-b2a3-4a67-b2fc-c251755ee766" (UID: "42768e93-b2a3-4a67-b2fc-c251755ee766"). InnerVolumeSpecName "kube-api-access-4qc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:41:06.496834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.496746 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qc2l\" (UniqueName: \"kubernetes.io/projected/42768e93-b2a3-4a67-b2fc-c251755ee766-kube-api-access-4qc2l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:06.496834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.496780 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:06.496834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:06.496790 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42768e93-b2a3-4a67-b2fc-c251755ee766-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:07.196649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:07.196617 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" Nov 03 04:41:07.196864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:07.196659 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j" event={"ID":"42768e93-b2a3-4a67-b2fc-c251755ee766","Type":"ContainerDied","Data":"6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc"} Nov 03 04:41:07.196864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:07.196707 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6641880b78ec090b9edfba006d3f1f24b10617c6b744acfd8eee373d592176cc" Nov 03 04:41:13.456341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.456293 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv"] Nov 03 04:41:13.456904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.456713 5275 cpu_manager.go:414] "containerMap: removing container" podUID="42768e93-b2a3-4a67-b2fc-c251755ee766" containerName="revoke-keys" Nov 03 04:41:13.456904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.456725 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42768e93-b2a3-4a67-b2fc-c251755ee766" containerName="revoke-keys" Nov 03 04:41:13.456904 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.456800 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="42768e93-b2a3-4a67-b2fc-c251755ee766" containerName="revoke-keys" Nov 03 04:41:13.459829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.459805 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.515553 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.515053 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:41:13.515821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.515071 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:41:13.515821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.515158 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:41:13.564808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.564775 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:41:13.565144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.565116 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.565256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.565240 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.565352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.565334 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7kt\" (UniqueName: \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.568188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.568159 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv"] Nov 03 04:41:13.666412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.666375 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.666596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.666450 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7kt\" (UniqueName: \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.666596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.666484 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.667273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.667238 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.668988 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.668958 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.727708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.727635 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7kt\" (UniqueName: \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\") pod \"maas-api-revoke-sub-d3fba056-tmvxv\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:13.777911 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:13.777830 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:14.165433 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:14.165399 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv"] Nov 03 04:41:14.167662 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:41:14.167635 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb676a060_e5be_45dd_87aa_cc9a85a2c49e.slice/crio-2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449 WatchSource:0}: Error finding container 2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449: Status 404 returned error can't find the container with id 2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449 Nov 03 04:41:14.235129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:14.235095 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" event={"ID":"b676a060-e5be-45dd-87aa-cc9a85a2c49e","Type":"ContainerStarted","Data":"2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449"} Nov 03 04:41:15.242078 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:15.242040 5275 generic.go:356] "Generic (PLEG): container finished" podID="b676a060-e5be-45dd-87aa-cc9a85a2c49e" containerID="111fb1abd6e70bb99f38c66f983fbf0d1eac35f941362009dace902f1e86b92b" exitCode=0 Nov 03 04:41:15.242484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:15.242126 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" event={"ID":"b676a060-e5be-45dd-87aa-cc9a85a2c49e","Type":"ContainerDied","Data":"111fb1abd6e70bb99f38c66f983fbf0d1eac35f941362009dace902f1e86b92b"} Nov 03 04:41:16.442209 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.441895 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:16.495167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.495118 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\") pod \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " Nov 03 04:41:16.495670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.495635 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca" pod "b676a060-e5be-45dd-87aa-cc9a85a2c49e" (UID: "b676a060-e5be-45dd-87aa-cc9a85a2c49e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:41:16.496350 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.496327 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\") pod \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " Nov 03 04:41:16.496470 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.496430 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\" (UniqueName: \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\") pod \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\" (UID: \"b676a060-e5be-45dd-87aa-cc9a85a2c49e\") " Nov 03 04:41:16.497437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.497405 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:16.511420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.511380 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt" pod "b676a060-e5be-45dd-87aa-cc9a85a2c49e" (UID: "b676a060-e5be-45dd-87aa-cc9a85a2c49e"). InnerVolumeSpecName "kube-api-access-2m7kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:41:16.511948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.511906 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca" pod "b676a060-e5be-45dd-87aa-cc9a85a2c49e" (UID: "b676a060-e5be-45dd-87aa-cc9a85a2c49e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:41:16.601899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.598256 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b676a060-e5be-45dd-87aa-cc9a85a2c49e-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:16.601899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:16.598299 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m7kt\" (UniqueName: \"kubernetes.io/projected/b676a060-e5be-45dd-87aa-cc9a85a2c49e-kube-api-access-2m7kt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:17.295605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:17.294334 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" event={"ID":"b676a060-e5be-45dd-87aa-cc9a85a2c49e","Type":"ContainerDied","Data":"2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449"} Nov 03 04:41:17.295605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:17.294405 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2915019c9767616712339791e1e64a5b5093e1faa8cfc8be68b14c5aab8bb449" Nov 03 04:41:17.295605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:17.294490 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv" Nov 03 04:41:18.460876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:18.459959 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:41:18.460876 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:18.460030 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:41:24.049239 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:24.049186 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w"] Nov 03 04:41:24.098810 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:24.098771 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-thn6w"] Nov 03 04:41:24.983282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:24.983244 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70afcc0-7844-4d66-86d3-f16145a105e0" path="/var/lib/kubelet/pods/a70afcc0-7844-4d66-86d3-f16145a105e0/volumes" Nov 03 04:41:32.344015 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:41:32.343968 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:41:48.569020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.568978 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l"] Nov 03 04:41:48.569635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.569614 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b676a060-e5be-45dd-87aa-cc9a85a2c49e" containerName="revoke-keys" Nov 03 04:41:48.569728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.569638 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b676a060-e5be-45dd-87aa-cc9a85a2c49e" containerName="revoke-keys" Nov 03 04:41:48.569787 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.569737 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b676a060-e5be-45dd-87aa-cc9a85a2c49e" containerName="revoke-keys" Nov 03 04:41:48.573106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.573085 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.631074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.631036 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l"] Nov 03 04:41:48.648688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.648648 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhv4\" (UniqueName: \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.648903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.648711 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.648903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.648811 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.677458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.677422 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:41:48.677458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.677435 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:41:48.677671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.677517 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:41:48.677671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.677661 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:41:48.750399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.750361 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhv4\" (UniqueName: \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.750609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.750429 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.750609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.750474 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.751094 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.751073 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.753109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.753082 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.811383 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.811348 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhv4\" (UniqueName: \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\") pod \"maas-api-revoke-sub-ae594979-kpv4l\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:48.886162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:48.886076 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:49.050231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:49.050191 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn"] Nov 03 04:41:49.052996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:49.052969 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce151bc8-w9tgn"] Nov 03 04:41:49.260028 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:49.259997 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l"] Nov 03 04:41:49.261949 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:41:49.261923 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a29a228_de1c_489c_be17_221960dd6600.slice/crio-220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f WatchSource:0}: Error finding container 220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f: Status 404 returned error can't find the container with id 220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f Nov 03 04:41:49.510644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:49.510554 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" event={"ID":"4a29a228-de1c-489c-be17-221960dd6600","Type":"ContainerStarted","Data":"220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f"} Nov 03 04:41:50.517784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:50.517698 5275 generic.go:356] "Generic (PLEG): container finished" podID="4a29a228-de1c-489c-be17-221960dd6600" containerID="56f9084f0dacd61fc25b224fc23f938adee3d84e76fd3703d02595f5cffd7c46" exitCode=0 Nov 03 04:41:50.518189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:50.517783 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" event={"ID":"4a29a228-de1c-489c-be17-221960dd6600","Type":"ContainerDied","Data":"56f9084f0dacd61fc25b224fc23f938adee3d84e76fd3703d02595f5cffd7c46"} Nov 03 04:41:50.613104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:50.613079 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:41:50.655222 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:50.655195 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:41:50.984537 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:50.984494 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7658d1ea-5dc7-464a-9e17-9ea279cc4aca" path="/var/lib/kubelet/pods/7658d1ea-5dc7-464a-9e17-9ea279cc4aca/volumes" Nov 03 04:41:51.661951 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.661926 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:51.780610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.780522 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\" (UniqueName: \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\") pod \"4a29a228-de1c-489c-be17-221960dd6600\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " Nov 03 04:41:51.780789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.780625 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\") pod \"4a29a228-de1c-489c-be17-221960dd6600\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " Nov 03 04:41:51.780789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.780673 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\") pod \"4a29a228-de1c-489c-be17-221960dd6600\" (UID: \"4a29a228-de1c-489c-be17-221960dd6600\") " Nov 03 04:41:51.781220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.781179 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca" pod "4a29a228-de1c-489c-be17-221960dd6600" (UID: "4a29a228-de1c-489c-be17-221960dd6600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:41:51.783172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.783144 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca" pod "4a29a228-de1c-489c-be17-221960dd6600" (UID: "4a29a228-de1c-489c-be17-221960dd6600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:41:51.783308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.783290 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4" pod "4a29a228-de1c-489c-be17-221960dd6600" (UID: "4a29a228-de1c-489c-be17-221960dd6600"). InnerVolumeSpecName "kube-api-access-rhhv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:41:51.881964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.881921 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhv4\" (UniqueName: \"kubernetes.io/projected/4a29a228-de1c-489c-be17-221960dd6600-kube-api-access-rhhv4\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:51.881964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.881956 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a29a228-de1c-489c-be17-221960dd6600-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:51.881964 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:51.881968 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a29a228-de1c-489c-be17-221960dd6600-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:41:52.530586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:52.530548 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" Nov 03 04:41:52.530586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:52.530562 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l" event={"ID":"4a29a228-de1c-489c-be17-221960dd6600","Type":"ContainerDied","Data":"220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f"} Nov 03 04:41:52.530807 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:52.530606 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220e6ff37eb78a32e8bafa398f10105173f2e7e4abe4c32e661a81df43dbd74f" Nov 03 04:41:55.398242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:41:55.398207 5275 scope.go:122] "RemoveContainer" containerID="3e51b4a098af387eb418ae0775621012c1ff3b17d1460c20f6a3fa937580b826" Nov 03 04:42:02.331505 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:42:02.331462 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:42:10.655307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.655272 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh"] Nov 03 04:42:10.655960 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.655939 5275 cpu_manager.go:414] "containerMap: removing container" podUID="4a29a228-de1c-489c-be17-221960dd6600" containerName="revoke-keys" Nov 03 04:42:10.656030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.655963 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a29a228-de1c-489c-be17-221960dd6600" containerName="revoke-keys" Nov 03 04:42:10.656110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.656098 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a29a228-de1c-489c-be17-221960dd6600" containerName="revoke-keys" Nov 03 04:42:10.659467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.659451 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.711828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.711676 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:42:10.746426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.746391 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.746634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.746456 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.746634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.746475 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjlg\" (UniqueName: \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.761235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.761201 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:42:10.761235 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.761227 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:42:10.761435 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.761213 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:42:10.763618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.763597 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh"] Nov 03 04:42:10.847779 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.847732 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.847983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.847809 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.847983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.847828 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjlg\" (UniqueName: \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.848406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.848381 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.850296 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.850272 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.911273 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.911198 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjlg\" (UniqueName: \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\") pod \"maas-api-revoke-sub-a4aef7d7-vhvrh\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:10.974888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:10.974833 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:11.354152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:11.354128 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh"] Nov 03 04:42:11.356417 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:42:11.356389 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e009273_d621_440f_958e_934a41f78aa2.slice/crio-1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179 WatchSource:0}: Error finding container 1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179: Status 404 returned error can't find the container with id 1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179 Nov 03 04:42:11.661675 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:11.661641 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" event={"ID":"9e009273-d621-440f-958e-934a41f78aa2","Type":"ContainerStarted","Data":"1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179"} Nov 03 04:42:12.669478 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:12.669441 5275 generic.go:356] "Generic (PLEG): container finished" podID="9e009273-d621-440f-958e-934a41f78aa2" containerID="ce60e63c2382346f3d8e96dcf10802e3c1323aca237eab9f668531a3a87e5798" exitCode=0 Nov 03 04:42:12.669916 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:12.669501 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" event={"ID":"9e009273-d621-440f-958e-934a41f78aa2","Type":"ContainerDied","Data":"ce60e63c2382346f3d8e96dcf10802e3c1323aca237eab9f668531a3a87e5798"} Nov 03 04:42:13.815141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.815115 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:13.877737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.877703 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\" (UniqueName: \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\") pod \"9e009273-d621-440f-958e-934a41f78aa2\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " Nov 03 04:42:13.877737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.877745 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\") pod \"9e009273-d621-440f-958e-934a41f78aa2\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " Nov 03 04:42:13.878040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.877818 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\") pod \"9e009273-d621-440f-958e-934a41f78aa2\" (UID: \"9e009273-d621-440f-958e-934a41f78aa2\") " Nov 03 04:42:13.878215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.878183 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca" pod "9e009273-d621-440f-958e-934a41f78aa2" (UID: "9e009273-d621-440f-958e-934a41f78aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:42:13.880115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.880085 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca" pod "9e009273-d621-440f-958e-934a41f78aa2" (UID: "9e009273-d621-440f-958e-934a41f78aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:42:13.880115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.880110 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg" pod "9e009273-d621-440f-958e-934a41f78aa2" (UID: "9e009273-d621-440f-958e-934a41f78aa2"). InnerVolumeSpecName "kube-api-access-qnjlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:42:13.978912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.978803 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjlg\" (UniqueName: \"kubernetes.io/projected/9e009273-d621-440f-958e-934a41f78aa2-kube-api-access-qnjlg\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:13.978912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.978887 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e009273-d621-440f-958e-934a41f78aa2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:13.978912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:13.978900 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e009273-d621-440f-958e-934a41f78aa2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:14.682935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:14.682903 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" Nov 03 04:42:14.683139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:14.682901 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh" event={"ID":"9e009273-d621-440f-958e-934a41f78aa2","Type":"ContainerDied","Data":"1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179"} Nov 03 04:42:14.683139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:14.683032 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7f75f988cd27c3ef0e58d4bda23ad331b068355ef4eed9bfbe90b5864b2179" Nov 03 04:42:31.817932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.814112 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52"] Nov 03 04:42:31.817932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.814756 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9e009273-d621-440f-958e-934a41f78aa2" containerName="revoke-keys" Nov 03 04:42:31.817932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.814775 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e009273-d621-440f-958e-934a41f78aa2" containerName="revoke-keys" Nov 03 04:42:31.817932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.814927 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e009273-d621-440f-958e-934a41f78aa2" containerName="revoke-keys" Nov 03 04:42:31.825683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.824992 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:31.867828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.867783 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:31.868574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.868548 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:31.868751 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.868734 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpxk\" (UniqueName: \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:31.925888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.923185 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52"] Nov 03 04:42:31.959884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.953208 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:42:31.959884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.953313 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:42:31.959884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.953362 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:42:31.959884 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.953452 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:42:32.000899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.976484 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.000899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.976557 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.000899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:31.976587 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpxk\" (UniqueName: \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.018902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:32.003403 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.018902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:32.010257 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.144729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:32.144603 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpxk\" (UniqueName: \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\") pod \"maas-api-revoke-sub-79805ca9-6lh52\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.196411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:32.196380 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:32.610871 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:42:32.608152 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:42:33.465871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:33.453815 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:42:33.465871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:33.453894 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:42:33.482089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:33.471865 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52"] Nov 03 04:42:33.842129 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:33.842087 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" event={"ID":"8c8ff723-519d-4c4d-b653-e8270ea15bfe","Type":"ContainerStarted","Data":"da473abf26ea990ba6edc2c995fbc579724f5ae412f868ca3806f935c7edf923"} Nov 03 04:42:34.849758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:34.849717 5275 generic.go:356] "Generic (PLEG): container finished" podID="8c8ff723-519d-4c4d-b653-e8270ea15bfe" containerID="1802bb9d6d0fe86c00c5f4fb7b951de83a3d7bcdda8f9344f4cf1f05e0cf8a67" exitCode=0 Nov 03 04:42:34.850216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:34.849804 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" event={"ID":"8c8ff723-519d-4c4d-b653-e8270ea15bfe","Type":"ContainerDied","Data":"1802bb9d6d0fe86c00c5f4fb7b951de83a3d7bcdda8f9344f4cf1f05e0cf8a67"} Nov 03 04:42:36.004410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.004384 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:36.070252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.070219 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\") pod \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " Nov 03 04:42:36.070438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.070292 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\" (UniqueName: \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\") pod \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " Nov 03 04:42:36.070438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.070347 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\") pod \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\" (UID: \"8c8ff723-519d-4c4d-b653-e8270ea15bfe\") " Nov 03 04:42:36.070617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.070592 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca" pod "8c8ff723-519d-4c4d-b653-e8270ea15bfe" (UID: "8c8ff723-519d-4c4d-b653-e8270ea15bfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:42:36.070739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.070716 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:36.072687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.072659 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca" pod "8c8ff723-519d-4c4d-b653-e8270ea15bfe" (UID: "8c8ff723-519d-4c4d-b653-e8270ea15bfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:42:36.072782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.072699 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk" pod "8c8ff723-519d-4c4d-b653-e8270ea15bfe" (UID: "8c8ff723-519d-4c4d-b653-e8270ea15bfe"). InnerVolumeSpecName "kube-api-access-nvpxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:42:36.172361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.172273 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpxk\" (UniqueName: \"kubernetes.io/projected/8c8ff723-519d-4c4d-b653-e8270ea15bfe-kube-api-access-nvpxk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:36.172361 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.172307 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ff723-519d-4c4d-b653-e8270ea15bfe-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:36.864066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.864034 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" Nov 03 04:42:36.864286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.864063 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52" event={"ID":"8c8ff723-519d-4c4d-b653-e8270ea15bfe","Type":"ContainerDied","Data":"da473abf26ea990ba6edc2c995fbc579724f5ae412f868ca3806f935c7edf923"} Nov 03 04:42:36.864286 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:36.864117 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da473abf26ea990ba6edc2c995fbc579724f5ae412f868ca3806f935c7edf923" Nov 03 04:42:49.289678 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.289639 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl"] Nov 03 04:42:49.290931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.290903 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8c8ff723-519d-4c4d-b653-e8270ea15bfe" containerName="revoke-keys" Nov 03 04:42:49.291089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.291075 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8ff723-519d-4c4d-b653-e8270ea15bfe" containerName="revoke-keys" Nov 03 04:42:49.310946 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.291288 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8ff723-519d-4c4d-b653-e8270ea15bfe" containerName="revoke-keys" Nov 03 04:42:49.321116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.321085 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.350869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.344172 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl"] Nov 03 04:42:49.433875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.431677 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.433875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.431774 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.433875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.431884 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqhh\" (UniqueName: \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.492877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.492274 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:42:49.496872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.496726 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:42:49.496872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.496823 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:42:49.497132 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.496894 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:42:49.533873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.533429 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.533873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.533531 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.533873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.533627 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqhh\" (UniqueName: \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.537870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.534760 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.559926 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.543583 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.607695 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.607614 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqhh\" (UniqueName: \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\") pod \"maas-api-revoke-sub-0b7e8edb-d2stl\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:49.674070 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:49.674033 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:50.472940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:50.471961 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:42:50.472940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:50.472058 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:42:50.694875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:50.690950 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl"] Nov 03 04:42:50.995872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:50.995584 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" event={"ID":"35619080-6b6f-4cb3-9252-a8383db634ac","Type":"ContainerStarted","Data":"aa9ace5928191cd28ed4d6b798bf7207dbfa188e42e8331bc64975438e7d1b97"} Nov 03 04:42:52.002757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:52.002724 5275 generic.go:356] "Generic (PLEG): container finished" podID="35619080-6b6f-4cb3-9252-a8383db634ac" containerID="da4a6b2326aeef56c4110477528c72a8b093fe9a642cc203d9d0aa57a5fc3556" exitCode=0 Nov 03 04:42:52.003162 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:52.002808 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" event={"ID":"35619080-6b6f-4cb3-9252-a8383db634ac","Type":"ContainerDied","Data":"da4a6b2326aeef56c4110477528c72a8b093fe9a642cc203d9d0aa57a5fc3556"} Nov 03 04:42:53.155607 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.155575 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:53.183723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.183688 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\") pod \"35619080-6b6f-4cb3-9252-a8383db634ac\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " Nov 03 04:42:53.183947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.183734 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\") pod \"35619080-6b6f-4cb3-9252-a8383db634ac\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " Nov 03 04:42:53.183947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.183771 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\" (UniqueName: \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\") pod \"35619080-6b6f-4cb3-9252-a8383db634ac\" (UID: \"35619080-6b6f-4cb3-9252-a8383db634ac\") " Nov 03 04:42:53.184179 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.184153 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca" pod "35619080-6b6f-4cb3-9252-a8383db634ac" (UID: "35619080-6b6f-4cb3-9252-a8383db634ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:42:53.186093 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.186066 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca" pod "35619080-6b6f-4cb3-9252-a8383db634ac" (UID: "35619080-6b6f-4cb3-9252-a8383db634ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:42:53.186185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.186164 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh" pod "35619080-6b6f-4cb3-9252-a8383db634ac" (UID: "35619080-6b6f-4cb3-9252-a8383db634ac"). InnerVolumeSpecName "kube-api-access-jdqhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:42:53.285320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.285287 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:53.285320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.285317 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35619080-6b6f-4cb3-9252-a8383db634ac-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:53.285320 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:53.285326 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdqhh\" (UniqueName: \"kubernetes.io/projected/35619080-6b6f-4cb3-9252-a8383db634ac-kube-api-access-jdqhh\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:42:54.015447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:54.015413 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" event={"ID":"35619080-6b6f-4cb3-9252-a8383db634ac","Type":"ContainerDied","Data":"aa9ace5928191cd28ed4d6b798bf7207dbfa188e42e8331bc64975438e7d1b97"} Nov 03 04:42:54.015636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:54.015467 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9ace5928191cd28ed4d6b798bf7207dbfa188e42e8331bc64975438e7d1b97" Nov 03 04:42:54.015636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:54.015429 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl" Nov 03 04:42:55.421732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:55.421699 5275 scope.go:122] "RemoveContainer" containerID="3231cd515adf2c29da3e8831fa7c342e4166b95983c732c5c6ea4018b4fa680d" Nov 03 04:42:55.431561 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:42:55.431541 5275 scope.go:122] "RemoveContainer" containerID="62bbbb6269f8d4891d3f8c19b36bd38d27172d6d5e41aada5ac09017aafdb572" Nov 03 04:43:00.625588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.625551 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr"] Nov 03 04:43:00.626036 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.626013 5275 cpu_manager.go:414] "containerMap: removing container" podUID="35619080-6b6f-4cb3-9252-a8383db634ac" containerName="revoke-keys" Nov 03 04:43:00.626036 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.626029 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35619080-6b6f-4cb3-9252-a8383db634ac" containerName="revoke-keys" Nov 03 04:43:00.626203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.626120 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="35619080-6b6f-4cb3-9252-a8383db634ac" containerName="revoke-keys" Nov 03 04:43:00.630638 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.630613 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.661589 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.661555 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.661760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.661597 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k629m\" (UniqueName: \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.661760 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.661686 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.734002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.733966 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:43:00.734264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.734003 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:43:00.734264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.734158 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:43:00.734264 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.733966 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:43:00.736667 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.736643 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr"] Nov 03 04:43:00.762452 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.762415 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.762622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.762460 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k629m\" (UniqueName: \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.762622 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.762497 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.763287 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.763212 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.765158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.765133 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.824439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.824398 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k629m\" (UniqueName: \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\") pod \"maas-api-revoke-sub-3d4156f1-7dwbr\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:00.944475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:00.944387 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:01.323799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:01.323766 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr"] Nov 03 04:43:01.325072 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:43:01.325041 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b1a5f6_af8e_4bce_af75_e2bc7623d1fb.slice/crio-44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76 WatchSource:0}: Error finding container 44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76: Status 404 returned error can't find the container with id 44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76 Nov 03 04:43:02.062269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:02.062232 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" event={"ID":"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb","Type":"ContainerStarted","Data":"b714349d39fb00f6c3912472d20d3839a5cf3cdeddeb1c3dc91fab4791a86266"} Nov 03 04:43:02.062724 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:02.062285 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" event={"ID":"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb","Type":"ContainerStarted","Data":"44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76"} Nov 03 04:43:02.179528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:02.179475 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" podStartSLOduration=2.179460005 podStartE2EDuration="2.179460005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:43:02.177912674 +0000 UTC m=+6073.330601094" watchObservedRunningTime="2026-11-03 04:43:02.179460005 +0000 UTC m=+6073.332148425" Nov 03 04:43:02.336307 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:43:02.336212 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:43:03.070190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:03.070154 5275 generic.go:356] "Generic (PLEG): container finished" podID="89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" containerID="b714349d39fb00f6c3912472d20d3839a5cf3cdeddeb1c3dc91fab4791a86266" exitCode=0 Nov 03 04:43:03.070619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:03.070252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" event={"ID":"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb","Type":"ContainerDied","Data":"b714349d39fb00f6c3912472d20d3839a5cf3cdeddeb1c3dc91fab4791a86266"} Nov 03 04:43:04.213484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.213454 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:04.302524 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.302491 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\" (UniqueName: \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\") pod \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " Nov 03 04:43:04.302743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.302586 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\") pod \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " Nov 03 04:43:04.302743 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.302628 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\") pod \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\" (UID: \"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb\") " Nov 03 04:43:04.303034 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.303000 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca" pod "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" (UID: "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:43:04.304840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.304809 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca" pod "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" (UID: "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:43:04.305002 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.304923 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m" pod "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" (UID: "89b1a5f6-af8e-4bce-af75-e2bc7623d1fb"). InnerVolumeSpecName "kube-api-access-k629m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:43:04.404481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.404395 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:04.404481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.404426 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:04.404481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:04.404437 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k629m\" (UniqueName: \"kubernetes.io/projected/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb-kube-api-access-k629m\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:05.083905 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:05.083874 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" Nov 03 04:43:05.084136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:05.083873 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr" event={"ID":"89b1a5f6-af8e-4bce-af75-e2bc7623d1fb","Type":"ContainerDied","Data":"44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76"} Nov 03 04:43:05.084136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:05.083999 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c4ef7c6a0f4cd7d1d8ad55d4ceb9cd5336a67724a06d899d92b93c243aff76" Nov 03 04:43:18.458507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:18.457943 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:43:18.458507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:18.458052 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:43:25.463066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.463026 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr"] Nov 03 04:43:25.512790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.512753 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dwbr"] Nov 03 04:43:25.571362 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.571327 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn"] Nov 03 04:43:25.571738 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.571726 5275 cpu_manager.go:414] "containerMap: removing container" podUID="89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" containerName="revoke-keys" Nov 03 04:43:25.571796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.571740 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" containerName="revoke-keys" Nov 03 04:43:25.571870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.571842 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" containerName="revoke-keys" Nov 03 04:43:25.576404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.576382 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.630825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.630783 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:43:25.631073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.630887 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:43:25.631073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.630943 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:43:25.631839 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.631013 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:43:25.635319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.635290 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn"] Nov 03 04:43:25.734748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.734653 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.734748 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.734703 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.735044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.734869 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb82s\" (UniqueName: \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.835978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.835944 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.836159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.835999 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.836159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.836079 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb82s\" (UniqueName: \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.836612 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.836591 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.838506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.838482 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:25.897203 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:25.897163 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb82s\" (UniqueName: \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\") pod \"maas-api-revoke-sub-3d4156f1-jx7nn\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:26.190032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:26.190000 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:26.565065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:26.565040 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn"] Nov 03 04:43:26.566903 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:43:26.566873 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c22f6_6c1b_4af3_8ea7_82b6f0579cf3.slice/crio-088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be WatchSource:0}: Error finding container 088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be: Status 404 returned error can't find the container with id 088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be Nov 03 04:43:26.984021 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:26.983978 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b1a5f6-af8e-4bce-af75-e2bc7623d1fb" path="/var/lib/kubelet/pods/89b1a5f6-af8e-4bce-af75-e2bc7623d1fb/volumes" Nov 03 04:43:27.276268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:27.276233 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" event={"ID":"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3","Type":"ContainerStarted","Data":"4b8cb19c0a1c7b348c902703a929fac4a5aeea499ce73cd5b8ce1f07407b3d7d"} Nov 03 04:43:27.276489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:27.276295 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" event={"ID":"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3","Type":"ContainerStarted","Data":"088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be"} Nov 03 04:43:27.392513 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:27.392457 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" podStartSLOduration=2.392440918 podStartE2EDuration="2.392440918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:43:27.391093112 +0000 UTC m=+6098.543781534" watchObservedRunningTime="2026-11-03 04:43:27.392440918 +0000 UTC m=+6098.545129351" Nov 03 04:43:28.284971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:28.284931 5275 generic.go:356] "Generic (PLEG): container finished" podID="261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" containerID="4b8cb19c0a1c7b348c902703a929fac4a5aeea499ce73cd5b8ce1f07407b3d7d" exitCode=0 Nov 03 04:43:28.285446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:28.285018 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" event={"ID":"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3","Type":"ContainerDied","Data":"4b8cb19c0a1c7b348c902703a929fac4a5aeea499ce73cd5b8ce1f07407b3d7d"} Nov 03 04:43:29.443642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.443612 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:29.474426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.474381 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\") pod \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " Nov 03 04:43:29.474643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.474461 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\" (UniqueName: \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\") pod \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " Nov 03 04:43:29.474643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.474537 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\") pod \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\" (UID: \"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3\") " Nov 03 04:43:29.475031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.475002 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca" pod "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" (UID: "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:43:29.476835 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.476805 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s" pod "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" (UID: "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3"). InnerVolumeSpecName "kube-api-access-nb82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:43:29.476986 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.476972 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca" pod "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" (UID: "261c22f6-6c1b-4af3-8ea7-82b6f0579cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:43:29.575974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.575874 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:29.575974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.575922 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb82s\" (UniqueName: \"kubernetes.io/projected/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-kube-api-access-nb82s\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:29.575974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:29.575938 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:43:30.298154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:30.298122 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" event={"ID":"261c22f6-6c1b-4af3-8ea7-82b6f0579cf3","Type":"ContainerDied","Data":"088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be"} Nov 03 04:43:30.298336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:30.298160 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn" Nov 03 04:43:30.298336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:30.298169 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088c9b3e29d2dceab0ecd7e1fe19bf684a4f9ef8b0377d99a05653a45e92d6be" Nov 03 04:43:32.333641 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:43:32.333588 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:43:55.481937 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:43:55.480982 5275 scope.go:122] "RemoveContainer" containerID="3198afe64a41b34dd8a886cfc4e734d700481895c2b871b6119b67f675e3614c" Nov 03 04:44:02.331718 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:44:02.331670 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:44:03.763357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.763313 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5"] Nov 03 04:44:03.764048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.764026 5275 cpu_manager.go:414] "containerMap: removing container" podUID="261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" containerName="revoke-keys" Nov 03 04:44:03.764115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.764051 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" containerName="revoke-keys" Nov 03 04:44:03.764156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.764148 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" containerName="revoke-keys" Nov 03 04:44:03.767928 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.767909 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.820704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.820670 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:44:03.820948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.820699 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:44:03.824966 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.824933 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5"] Nov 03 04:44:03.859367 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.859328 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtzj\" (UniqueName: \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.859562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.859387 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.859562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.859474 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.870397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.870368 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:44:03.870397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.870381 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:44:03.960194 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.960151 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtzj\" (UniqueName: \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.960421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.960217 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.960421 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.960246 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.960965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.960941 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:03.962778 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:03.962757 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:04.021254 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:04.021227 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtzj\" (UniqueName: \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\") pod \"maas-api-revoke-sub-e2fc9c5b-z5vn5\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:04.080696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:04.080650 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:04.455457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:04.455434 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5"] Nov 03 04:44:04.457593 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:44:04.457560 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f1c05a_8012_4f3b_bb9d_d7102932f5f8.slice/crio-f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf WatchSource:0}: Error finding container f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf: Status 404 returned error can't find the container with id f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf Nov 03 04:44:04.595888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:04.595834 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" event={"ID":"04f1c05a-8012-4f3b-bb9d-d7102932f5f8","Type":"ContainerStarted","Data":"f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf"} Nov 03 04:44:05.602593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:05.602499 5275 generic.go:356] "Generic (PLEG): container finished" podID="04f1c05a-8012-4f3b-bb9d-d7102932f5f8" containerID="23f5416c55187dbc71b09894b81f8f1a221b3331f9e795f91cf3807cacb0c6be" exitCode=0 Nov 03 04:44:05.602998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:05.602585 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" event={"ID":"04f1c05a-8012-4f3b-bb9d-d7102932f5f8","Type":"ContainerDied","Data":"23f5416c55187dbc71b09894b81f8f1a221b3331f9e795f91cf3807cacb0c6be"} Nov 03 04:44:06.743771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.743729 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:06.788081 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.788039 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\" (UniqueName: \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\") pod \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " Nov 03 04:44:06.788288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.788148 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\") pod \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " Nov 03 04:44:06.788288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.788213 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\") pod \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\" (UID: \"04f1c05a-8012-4f3b-bb9d-d7102932f5f8\") " Nov 03 04:44:06.788602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.788577 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca" pod "04f1c05a-8012-4f3b-bb9d-d7102932f5f8" (UID: "04f1c05a-8012-4f3b-bb9d-d7102932f5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:44:06.790424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.790395 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca" pod "04f1c05a-8012-4f3b-bb9d-d7102932f5f8" (UID: "04f1c05a-8012-4f3b-bb9d-d7102932f5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:44:06.790520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.790439 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj" pod "04f1c05a-8012-4f3b-bb9d-d7102932f5f8" (UID: "04f1c05a-8012-4f3b-bb9d-d7102932f5f8"). InnerVolumeSpecName "kube-api-access-cxtzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:44:06.889573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.889474 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:06.889573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.889522 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:06.889573 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:06.889539 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxtzj\" (UniqueName: \"kubernetes.io/projected/04f1c05a-8012-4f3b-bb9d-d7102932f5f8-kube-api-access-cxtzj\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:07.615399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:07.615371 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" Nov 03 04:44:07.615591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:07.615408 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5" event={"ID":"04f1c05a-8012-4f3b-bb9d-d7102932f5f8","Type":"ContainerDied","Data":"f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf"} Nov 03 04:44:07.615591 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:07.615455 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b12d537c4aea42079d8e0772048566a3f0832dbbe75d68297da96bc3a9bebf" Nov 03 04:44:14.562680 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.562641 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs"] Nov 03 04:44:14.563278 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.563257 5275 cpu_manager.go:414] "containerMap: removing container" podUID="04f1c05a-8012-4f3b-bb9d-d7102932f5f8" containerName="revoke-keys" Nov 03 04:44:14.563372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.563280 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f1c05a-8012-4f3b-bb9d-d7102932f5f8" containerName="revoke-keys" Nov 03 04:44:14.563434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.563413 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f1c05a-8012-4f3b-bb9d-d7102932f5f8" containerName="revoke-keys" Nov 03 04:44:14.566816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.566787 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.621648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.621613 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:44:14.666940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.666897 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rbp\" (UniqueName: \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.666940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.666944 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.667224 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.667096 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.671114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.671087 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:44:14.671285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.671192 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:44:14.671406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.671386 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:44:14.673914 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.673888 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs"] Nov 03 04:44:14.768309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.768271 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.768516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.768390 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rbp\" (UniqueName: \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.768516 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.768419 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.769208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.769137 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.770900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.770880 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.835409 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.835331 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rbp\" (UniqueName: \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\") pod \"maas-api-revoke-sub-5a29efb3-htcfs\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:14.883073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:14.883032 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:15.260026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:15.260001 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs"] Nov 03 04:44:15.262066 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:44:15.262034 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbd200e_dfc5_4543_8735_45876a004d63.slice/crio-c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449 WatchSource:0}: Error finding container c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449: Status 404 returned error can't find the container with id c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449 Nov 03 04:44:15.661576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:15.661532 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" event={"ID":"8bbd200e-dfc5-4543-8735-45876a004d63","Type":"ContainerStarted","Data":"a28470d30eff8fcb161d96d657482776b615f0b2b9e7be84b1701a96575f4089"} Nov 03 04:44:15.662003 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:15.661592 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" event={"ID":"8bbd200e-dfc5-4543-8735-45876a004d63","Type":"ContainerStarted","Data":"c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449"} Nov 03 04:44:15.779620 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:15.779559 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" podStartSLOduration=1.779541591 podStartE2EDuration="1.779541591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:44:15.779044553 +0000 UTC m=+6146.931732992" watchObservedRunningTime="2026-11-03 04:44:15.779541591 +0000 UTC m=+6146.932230012" Nov 03 04:44:16.668708 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:16.668676 5275 generic.go:356] "Generic (PLEG): container finished" podID="8bbd200e-dfc5-4543-8735-45876a004d63" containerID="a28470d30eff8fcb161d96d657482776b615f0b2b9e7be84b1701a96575f4089" exitCode=0 Nov 03 04:44:16.669223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:16.668728 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" event={"ID":"8bbd200e-dfc5-4543-8735-45876a004d63","Type":"ContainerDied","Data":"a28470d30eff8fcb161d96d657482776b615f0b2b9e7be84b1701a96575f4089"} Nov 03 04:44:17.934431 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:17.934342 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:18.008666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.008611 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\" (UniqueName: \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\") pod \"8bbd200e-dfc5-4543-8735-45876a004d63\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " Nov 03 04:44:18.008666 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.008672 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\") pod \"8bbd200e-dfc5-4543-8735-45876a004d63\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " Nov 03 04:44:18.009000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.008736 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\") pod \"8bbd200e-dfc5-4543-8735-45876a004d63\" (UID: \"8bbd200e-dfc5-4543-8735-45876a004d63\") " Nov 03 04:44:18.009279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.009242 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca" pod "8bbd200e-dfc5-4543-8735-45876a004d63" (UID: "8bbd200e-dfc5-4543-8735-45876a004d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:44:18.012118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.012073 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca" pod "8bbd200e-dfc5-4543-8735-45876a004d63" (UID: "8bbd200e-dfc5-4543-8735-45876a004d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:44:18.012430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.012402 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp" pod "8bbd200e-dfc5-4543-8735-45876a004d63" (UID: "8bbd200e-dfc5-4543-8735-45876a004d63"). InnerVolumeSpecName "kube-api-access-b8rbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:44:18.110055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.110018 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:18.110055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.110053 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rbp\" (UniqueName: \"kubernetes.io/projected/8bbd200e-dfc5-4543-8735-45876a004d63-kube-api-access-b8rbp\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:18.110280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.110066 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbd200e-dfc5-4543-8735-45876a004d63-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:18.708873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.708276 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" event={"ID":"8bbd200e-dfc5-4543-8735-45876a004d63","Type":"ContainerDied","Data":"c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449"} Nov 03 04:44:18.708873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.708345 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05865cb7779ce0813a5472e5c6177531862758ecbeeb4b8ca6c6155387de449" Nov 03 04:44:18.708873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:18.708418 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs" Nov 03 04:44:20.052873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:20.052684 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm"] Nov 03 04:44:20.164466 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:20.164428 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jl2tm"] Nov 03 04:44:20.982958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:20.982927 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0" path="/var/lib/kubelet/pods/3ad4f8cc-8ef5-4af7-afc6-b967ee4419e0/volumes" Nov 03 04:44:31.898530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:31.898494 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn"] Nov 03 04:44:31.947982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:31.947944 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jx7nn"] Nov 03 04:44:32.018673 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.018640 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2"] Nov 03 04:44:32.019098 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.019085 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8bbd200e-dfc5-4543-8735-45876a004d63" containerName="revoke-keys" Nov 03 04:44:32.019173 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.019100 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbd200e-dfc5-4543-8735-45876a004d63" containerName="revoke-keys" Nov 03 04:44:32.019216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.019199 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbd200e-dfc5-4543-8735-45876a004d63" containerName="revoke-keys" Nov 03 04:44:32.022683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.022666 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.077544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.077506 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:44:32.077802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.077777 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:44:32.078055 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.077790 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:44:32.078309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.078283 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:44:32.081867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.081824 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2"] Nov 03 04:44:32.094974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.094942 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.095154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.094981 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.095154 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.095015 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rpt\" (UniqueName: \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.196670 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.196580 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rpt\" (UniqueName: \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.196865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.196699 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.196865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.196722 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.197399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.197378 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.199300 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.199277 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.257725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.257686 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rpt\" (UniqueName: \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\") pod \"maas-api-revoke-sub-3d4156f1-vv9t2\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.328994 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:44:32.328954 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:44:32.335770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.335741 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:32.723473 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.723398 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2"] Nov 03 04:44:32.726171 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:44:32.726128 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519b6cd5_2458_429f_8220_053cca604251.slice/crio-e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d WatchSource:0}: Error finding container e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d: Status 404 returned error can't find the container with id e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d Nov 03 04:44:32.816859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.816805 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" event={"ID":"519b6cd5-2458-429f-8220-053cca604251","Type":"ContainerStarted","Data":"e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d"} Nov 03 04:44:32.983064 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:32.982993 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c22f6-6c1b-4af3-8ea7-82b6f0579cf3" path="/var/lib/kubelet/pods/261c22f6-6c1b-4af3-8ea7-82b6f0579cf3/volumes" Nov 03 04:44:33.824646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:33.824610 5275 generic.go:356] "Generic (PLEG): container finished" podID="519b6cd5-2458-429f-8220-053cca604251" containerID="33fdfdc1c06ecd2f80be519a65b7e71fee66ce9fc4d08f021b3468db567e278c" exitCode=0 Nov 03 04:44:33.825045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:33.824689 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" event={"ID":"519b6cd5-2458-429f-8220-053cca604251","Type":"ContainerDied","Data":"33fdfdc1c06ecd2f80be519a65b7e71fee66ce9fc4d08f021b3468db567e278c"} Nov 03 04:44:34.972450 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:34.972411 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:35.127789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.127701 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\") pod \"519b6cd5-2458-429f-8220-053cca604251\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " Nov 03 04:44:35.127989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.127833 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\" (UniqueName: \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\") pod \"519b6cd5-2458-429f-8220-053cca604251\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " Nov 03 04:44:35.127989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.127984 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\") pod \"519b6cd5-2458-429f-8220-053cca604251\" (UID: \"519b6cd5-2458-429f-8220-053cca604251\") " Nov 03 04:44:35.128221 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.128185 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca" pod "519b6cd5-2458-429f-8220-053cca604251" (UID: "519b6cd5-2458-429f-8220-053cca604251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:44:35.130131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.130106 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca" pod "519b6cd5-2458-429f-8220-053cca604251" (UID: "519b6cd5-2458-429f-8220-053cca604251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:44:35.130284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.130271 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt" pod "519b6cd5-2458-429f-8220-053cca604251" (UID: "519b6cd5-2458-429f-8220-053cca604251"). InnerVolumeSpecName "kube-api-access-98rpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:44:35.229468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.229426 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519b6cd5-2458-429f-8220-053cca604251-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:35.229468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.229464 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519b6cd5-2458-429f-8220-053cca604251-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:35.229468 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.229474 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rpt\" (UniqueName: \"kubernetes.io/projected/519b6cd5-2458-429f-8220-053cca604251-kube-api-access-98rpt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:44:35.836605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.836575 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" Nov 03 04:44:35.836769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.836609 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2" event={"ID":"519b6cd5-2458-429f-8220-053cca604251","Type":"ContainerDied","Data":"e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d"} Nov 03 04:44:35.836769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:35.836665 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9674ce9b36f23a2cfd814627f2c0412e9d35932dcf6aa4621777a3df9990b8d" Nov 03 04:44:48.449875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.449599 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:44:48.449875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.449761 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:44:48.737935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.736833 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:44:48.737935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.737655 5275 cpu_manager.go:414] "containerMap: removing container" podUID="519b6cd5-2458-429f-8220-053cca604251" containerName="revoke-keys" Nov 03 04:44:48.737935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.737680 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519b6cd5-2458-429f-8220-053cca604251" containerName="revoke-keys" Nov 03 04:44:48.737935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.737894 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="519b6cd5-2458-429f-8220-053cca604251" containerName="revoke-keys" Nov 03 04:44:48.742546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.742515 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:48.809888 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.795832 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bcb427-openshift-default-dockercfg-25fvx\"" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.955923 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969053 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969113 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969144 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969200 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969229 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969289 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969330 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969370 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpb7l\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.018045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:48.969413 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165124 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165236 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165275 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165307 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165363 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165395 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165460 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165498 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.165540 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpb7l\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.178925 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.167764 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.194671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.194621 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.195814 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.195758 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.196378 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.196349 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.199242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.199205 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.357869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.356428 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.357869 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.356640 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.364588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.364515 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.380875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.369784 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpb7l\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\") pod \"e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:49.568387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.559272 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bcb427-openshift-default-dockercfg-25fvx\"" Nov 03 04:44:49.568387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:49.567045 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:50.084116 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:50.084088 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:44:50.085032 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:44:50.085009 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660a2a7c_9766_4196_bac7_082eaaabcac8.slice/crio-2e0ead7cce8116cf121edbcae62f0b89fbb390b0cf9ce8376e48af9ed15be98c WatchSource:0}: Error finding container 2e0ead7cce8116cf121edbcae62f0b89fbb390b0cf9ce8376e48af9ed15be98c: Status 404 returned error can't find the container with id 2e0ead7cce8116cf121edbcae62f0b89fbb390b0cf9ce8376e48af9ed15be98c Nov 03 04:44:50.088011 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:50.087917 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:44:50.088358 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:50.088322 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:44:50.095401 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:50.095321 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:44:51.047335 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:51.047292 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerStarted","Data":"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066"} Nov 03 04:44:51.047775 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:51.047351 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerStarted","Data":"2e0ead7cce8116cf121edbcae62f0b89fbb390b0cf9ce8376e48af9ed15be98c"} Nov 03 04:44:51.220623 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:51.220552 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podStartSLOduration=3.220533957 podStartE2EDuration="3.220533957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:44:51.217112658 +0000 UTC m=+6182.369801078" watchObservedRunningTime="2026-11-03 04:44:51.220533957 +0000 UTC m=+6182.373222379" Nov 03 04:44:51.567294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:51.567258 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:51.572475 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:51.572445 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:52.051921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:52.051887 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f"] Nov 03 04:44:52.054225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:52.054196 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bg94f"] Nov 03 04:44:52.054418 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:52.054384 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:52.055509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:52.055487 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:44:52.983736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:52.983699 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bbd37e-1bbc-4ac9-9e06-946f16285046" path="/var/lib/kubelet/pods/73bbd37e-1bbc-4ac9-9e06-946f16285046/volumes" Nov 03 04:44:57.900201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:57.900163 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:44:57.905114 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:57.905095 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:57.958283 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:57.958241 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bcb427\"" Nov 03 04:44:58.010580 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.010542 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:44:58.018657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.018611 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzrf\" (UniqueName: \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.018936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.018672 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.120352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.120301 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzrf\" (UniqueName: \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.120548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.120358 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.121208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.121179 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.181927 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.181819 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzrf\" (UniqueName: \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\") pod \"payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.219972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.219932 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:58.599432 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:58.599407 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:44:58.601093 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:44:58.601066 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd03194d_2810_4ddd_a614_4d1330517347.slice/crio-9678a056d7ad9ea6b821e19af5a8e9ccc0b4733db7f2c738d6de09524d492d8d WatchSource:0}: Error finding container 9678a056d7ad9ea6b821e19af5a8e9ccc0b4733db7f2c738d6de09524d492d8d: Status 404 returned error can't find the container with id 9678a056d7ad9ea6b821e19af5a8e9ccc0b4733db7f2c738d6de09524d492d8d Nov 03 04:44:59.096263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:59.096211 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" event={"ID":"bd03194d-2810-4ddd-a614-4d1330517347","Type":"ContainerStarted","Data":"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633"} Nov 03 04:44:59.096899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:59.096280 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" event={"ID":"bd03194d-2810-4ddd-a614-4d1330517347","Type":"ContainerStarted","Data":"9678a056d7ad9ea6b821e19af5a8e9ccc0b4733db7f2c738d6de09524d492d8d"} Nov 03 04:44:59.096899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:59.096327 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:44:59.216568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:44:59.216501 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" podStartSLOduration=2.216480113 podStartE2EDuration="2.216480113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:44:59.212138192 +0000 UTC m=+6190.364826614" watchObservedRunningTime="2026-11-03 04:44:59.216480113 +0000 UTC m=+6190.369168535" Nov 03 04:45:00.200555 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.200518 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:00.204340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.204319 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.237284 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.237244 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:00.242410 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.242372 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrtv\" (UniqueName: \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.242635 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.242525 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.259488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.259456 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:45:00.259674 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.259468 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:45:00.259674 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.259534 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:45:00.343581 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.343546 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrtv\" (UniqueName: \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.343768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.343623 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.344298 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.344279 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.404998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.404959 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrtv\" (UniqueName: \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\") pod \"maas-api-key-cleanup-29894685-nt4cm\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.517747 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.517661 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:00.903365 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:00.903334 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:00.906487 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:45:00.906455 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644412fd_c2f3_4011_9824_1ff1cf84e086.slice/crio-e6fcad51788967093e542c72d1086a2bcfc2bddf865a33fd8d6aef07a38bf91c WatchSource:0}: Error finding container e6fcad51788967093e542c72d1086a2bcfc2bddf865a33fd8d6aef07a38bf91c: Status 404 returned error can't find the container with id e6fcad51788967093e542c72d1086a2bcfc2bddf865a33fd8d6aef07a38bf91c Nov 03 04:45:01.110259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:01.110224 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" event={"ID":"644412fd-c2f3-4011-9824-1ff1cf84e086","Type":"ContainerStarted","Data":"e6fcad51788967093e542c72d1086a2bcfc2bddf865a33fd8d6aef07a38bf91c"} Nov 03 04:45:02.122368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:02.122336 5275 generic.go:356] "Generic (PLEG): container finished" podID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerID="b934a451dde4d1ebc96e79cd7ae8dd012e974bd6668433e8f02cb370c7b97c1f" exitCode=60 Nov 03 04:45:02.122770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:02.122411 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" event={"ID":"644412fd-c2f3-4011-9824-1ff1cf84e086","Type":"ContainerDied","Data":"b934a451dde4d1ebc96e79cd7ae8dd012e974bd6668433e8f02cb370c7b97c1f"} Nov 03 04:45:02.122770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:02.122643 5275 scope.go:122] "RemoveContainer" containerID="b934a451dde4d1ebc96e79cd7ae8dd012e974bd6668433e8f02cb370c7b97c1f" Nov 03 04:45:02.333922 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:45:02.333882 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:45:03.131000 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:03.130964 5275 generic.go:356] "Generic (PLEG): container finished" podID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerID="0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257" exitCode=60 Nov 03 04:45:03.131584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:03.131051 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" event={"ID":"644412fd-c2f3-4011-9824-1ff1cf84e086","Type":"ContainerDied","Data":"0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257"} Nov 03 04:45:03.131584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:03.131110 5275 scope.go:122] "RemoveContainer" containerID="b934a451dde4d1ebc96e79cd7ae8dd012e974bd6668433e8f02cb370c7b97c1f" Nov 03 04:45:03.131584 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:03.131409 5275 scope.go:122] "RemoveContainer" containerID="0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257" Nov 03 04:45:03.131795 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:45:03.131743 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894685-nt4cm_odh-ai-gateway-infra(644412fd-c2f3-4011-9824-1ff1cf84e086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" Nov 03 04:45:04.138963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:04.138919 5275 scope.go:122] "RemoveContainer" containerID="0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257" Nov 03 04:45:04.139406 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:45:04.139179 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894685-nt4cm_odh-ai-gateway-infra(644412fd-c2f3-4011-9824-1ff1cf84e086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" Nov 03 04:45:10.105293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:10.105257 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:45:15.978110 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:15.978076 5275 scope.go:122] "RemoveContainer" containerID="0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257" Nov 03 04:45:17.233210 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:17.233176 5275 generic.go:356] "Generic (PLEG): container finished" podID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerID="2beb3ee7dc0762c85a8ece5af4a9bf34a20d0a2559846555a4fbf21f1d3e3794" exitCode=60 Nov 03 04:45:17.233628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:17.233263 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" event={"ID":"644412fd-c2f3-4011-9824-1ff1cf84e086","Type":"ContainerDied","Data":"2beb3ee7dc0762c85a8ece5af4a9bf34a20d0a2559846555a4fbf21f1d3e3794"} Nov 03 04:45:17.233628 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:17.233347 5275 scope.go:122] "RemoveContainer" containerID="0b097d3f5245e7dba9c48cb629110af58ed4b4851770673e7a15d8ed42e02257" Nov 03 04:45:17.233756 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:17.233739 5275 scope.go:122] "RemoveContainer" containerID="2beb3ee7dc0762c85a8ece5af4a9bf34a20d0a2559846555a4fbf21f1d3e3794" Nov 03 04:45:17.234087 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:45:17.234065 5275 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894685-nt4cm_odh-ai-gateway-infra(644412fd-c2f3-4011-9824-1ff1cf84e086)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" Nov 03 04:45:18.055867 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.055746 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv"] Nov 03 04:45:18.103800 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.103768 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gnmqv"] Nov 03 04:45:18.240182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.240147 5275 generic.go:356] "Generic (PLEG): container finished" podID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerID="9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066" exitCode=0 Nov 03 04:45:18.240633 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.240223 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerDied","Data":"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066"} Nov 03 04:45:18.240704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.240662 5275 scope.go:122] "RemoveContainer" containerID="9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066" Nov 03 04:45:18.242030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.241992 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:45:18.242157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.242081 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:45:18.242157 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.242131 5275 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 04:45:18.367434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.367006 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:18.555079 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.555047 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:18.563123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.563052 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\") pod \"644412fd-c2f3-4011-9824-1ff1cf84e086\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " Nov 03 04:45:18.563269 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.563216 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\" (UniqueName: \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\") pod \"644412fd-c2f3-4011-9824-1ff1cf84e086\" (UID: \"644412fd-c2f3-4011-9824-1ff1cf84e086\") " Nov 03 04:45:18.563447 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.563423 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca" pod "644412fd-c2f3-4011-9824-1ff1cf84e086" (UID: "644412fd-c2f3-4011-9824-1ff1cf84e086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:45:18.563564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.563546 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644412fd-c2f3-4011-9824-1ff1cf84e086-maas-api-service-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:45:18.565480 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.565459 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv" pod "644412fd-c2f3-4011-9824-1ff1cf84e086" (UID: "644412fd-c2f3-4011-9824-1ff1cf84e086"). InnerVolumeSpecName "kube-api-access-kbrtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:45:18.664086 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.664049 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrtv\" (UniqueName: \"kubernetes.io/projected/644412fd-c2f3-4011-9824-1ff1cf84e086-kube-api-access-kbrtv\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:45:18.983113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:18.983013 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4848c4f-e8a0-4813-8854-aa59cbf524d9" path="/var/lib/kubelet/pods/c4848c4f-e8a0-4813-8854-aa59cbf524d9/volumes" Nov 03 04:45:19.273935 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.271622 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerStarted","Data":"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d"} Nov 03 04:45:19.275293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.275265 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" Nov 03 04:45:19.275562 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.275274 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm" event={"ID":"644412fd-c2f3-4011-9824-1ff1cf84e086","Type":"ContainerDied","Data":"e6fcad51788967093e542c72d1086a2bcfc2bddf865a33fd8d6aef07a38bf91c"} Nov 03 04:45:19.275664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.275624 5275 scope.go:122] "RemoveContainer" containerID="2beb3ee7dc0762c85a8ece5af4a9bf34a20d0a2559846555a4fbf21f1d3e3794" Nov 03 04:45:19.512832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.512784 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:19.516922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.516893 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894685-nt4cm"] Nov 03 04:45:19.569931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.569351 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:45:19.569931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.569488 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body= Nov 03 04:45:19.569931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:19.569531 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" Nov 03 04:45:20.573792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:20.573304 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body= Nov 03 04:45:20.573792 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:20.573372 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" Nov 03 04:45:20.985152 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:20.985061 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" path="/var/lib/kubelet/pods/644412fd-c2f3-4011-9824-1ff1cf84e086/volumes" Nov 03 04:45:21.571892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:21.571320 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body= Nov 03 04:45:21.571892 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:21.571393 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" Nov 03 04:45:22.058812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:22.057959 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:45:22.572873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:22.571258 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body= Nov 03 04:45:22.572873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:22.571320 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" Nov 03 04:45:23.569789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:23.569298 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body= Nov 03 04:45:23.569789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:23.569367 5275 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" Nov 03 04:45:24.572929 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:24.572896 5275 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:45:24.574115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:24.574089 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:45:31.050063 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:31.050025 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj"] Nov 03 04:45:31.099798 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:31.099762 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s8wvj"] Nov 03 04:45:32.332573 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:45:32.332537 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:45:32.983192 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:32.983153 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e812cad-c99d-4b40-9c2c-84da1580dbb5" path="/var/lib/kubelet/pods/4e812cad-c99d-4b40-9c2c-84da1580dbb5/volumes" Nov 03 04:45:55.672991 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:55.672956 5275 scope.go:122] "RemoveContainer" containerID="a76a45064e88e1704ef0e0a0a6412f6cf2f210fa927475ca1695f4e4439026a7" Nov 03 04:45:55.682619 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:45:55.682597 5275 scope.go:122] "RemoveContainer" containerID="23c84c841efbeb286187d278223f1cf56475c20190858e6f50f13bd2392eb7cd" Nov 03 04:46:02.333377 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:46:02.333334 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:46:07.050085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.050019 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j"] Nov 03 04:46:07.099436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.099399 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vwp4j"] Nov 03 04:46:07.614649 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.614608 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr"] Nov 03 04:46:07.615092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615077 5275 cpu_manager.go:414] "containerMap: removing container" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.615150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615094 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.615150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615103 5275 cpu_manager.go:414] "containerMap: removing container" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.615150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615109 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.615268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615199 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.615268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.615214 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:46:07.618350 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.618333 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.676808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.676773 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr"] Nov 03 04:46:07.696047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.695965 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.696236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.696115 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrv57\" (UniqueName: \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.696236 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.696152 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.722750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.722718 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:46:07.722750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.722717 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:46:07.723031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.722792 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:46:07.723031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.722899 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:46:07.796936 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.796900 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrv57\" (UniqueName: \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.797175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.796953 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.797175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.797046 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.797640 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.797605 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.799547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.799517 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.858220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.858188 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrv57\" (UniqueName: \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\") pod \"maas-api-revoke-sub-5520a9f4-8xkpr\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:07.932308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:07.932209 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:08.307983 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:08.307956 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr"] Nov 03 04:46:08.310446 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:46:08.310409 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa36d9f_6eca_4092_b119_1483b7e0d1ba.slice/crio-1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f WatchSource:0}: Error finding container 1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f: Status 404 returned error can't find the container with id 1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f Nov 03 04:46:08.633349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:08.633314 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" event={"ID":"5aa36d9f-6eca-4092-b119-1483b7e0d1ba","Type":"ContainerStarted","Data":"1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f"} Nov 03 04:46:08.983494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:08.983419 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42768e93-b2a3-4a67-b2fc-c251755ee766" path="/var/lib/kubelet/pods/42768e93-b2a3-4a67-b2fc-c251755ee766/volumes" Nov 03 04:46:09.640821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:09.640790 5275 generic.go:356] "Generic (PLEG): container finished" podID="5aa36d9f-6eca-4092-b119-1483b7e0d1ba" containerID="c5fecdf87f6d56f4c4d80a0f6b692986bc0a54332cd3b0349fcfa57105c11394" exitCode=0 Nov 03 04:46:09.641242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:09.640839 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" event={"ID":"5aa36d9f-6eca-4092-b119-1483b7e0d1ba","Type":"ContainerDied","Data":"c5fecdf87f6d56f4c4d80a0f6b692986bc0a54332cd3b0349fcfa57105c11394"} Nov 03 04:46:10.786187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.786157 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:10.929971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.929881 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\") pod \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " Nov 03 04:46:10.930136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.930027 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\" (UniqueName: \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\") pod \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " Nov 03 04:46:10.930136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.930129 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\") pod \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\" (UID: \"5aa36d9f-6eca-4092-b119-1483b7e0d1ba\") " Nov 03 04:46:10.930374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.930344 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca" pod "5aa36d9f-6eca-4092-b119-1483b7e0d1ba" (UID: "5aa36d9f-6eca-4092-b119-1483b7e0d1ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:46:10.930548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.930528 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:10.932451 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.932419 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca" pod "5aa36d9f-6eca-4092-b119-1483b7e0d1ba" (UID: "5aa36d9f-6eca-4092-b119-1483b7e0d1ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:46:10.932451 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:10.932425 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57" pod "5aa36d9f-6eca-4092-b119-1483b7e0d1ba" (UID: "5aa36d9f-6eca-4092-b119-1483b7e0d1ba"). InnerVolumeSpecName "kube-api-access-jrv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:46:11.031458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:11.031422 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrv57\" (UniqueName: \"kubernetes.io/projected/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-kube-api-access-jrv57\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:11.031458 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:11.031452 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa36d9f-6eca-4092-b119-1483b7e0d1ba-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:11.653566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:11.653532 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" Nov 03 04:46:11.653777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:11.653531 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr" event={"ID":"5aa36d9f-6eca-4092-b119-1483b7e0d1ba","Type":"ContainerDied","Data":"1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f"} Nov 03 04:46:11.653777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:11.653674 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ccfd6718d825ab7a23e74863e8d7b8c66d5174236952e4cdce29336cfd50a7f" Nov 03 04:46:17.052457 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:17.052414 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv"] Nov 03 04:46:17.101617 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:17.101582 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tmvxv"] Nov 03 04:46:18.983840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:18.983804 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b676a060-e5be-45dd-87aa-cc9a85a2c49e" path="/var/lib/kubelet/pods/b676a060-e5be-45dd-87aa-cc9a85a2c49e/volumes" Nov 03 04:46:25.609445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:25.609411 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:46:30.611231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:30.611174 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" containerID="cri-o://de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633" gracePeriod=25 Nov 03 04:46:32.341016 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:46:32.340974 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:46:40.103369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:40.103329 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:46:50.103547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.103504 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 04:46:50.656216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.656191 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:46:50.697587 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.697561 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:46:50.866614 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.866581 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:46:50.885159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.885074 5275 generic.go:356] "Generic (PLEG): container finished" podID="bd03194d-2810-4ddd-a614-4d1330517347" containerID="de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633" exitCode=0 Nov 03 04:46:50.885159 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.885121 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" Nov 03 04:46:50.885397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.885157 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" event={"ID":"bd03194d-2810-4ddd-a614-4d1330517347","Type":"ContainerDied","Data":"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633"} Nov 03 04:46:50.885397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.885252 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg" event={"ID":"bd03194d-2810-4ddd-a614-4d1330517347","Type":"ContainerDied","Data":"9678a056d7ad9ea6b821e19af5a8e9ccc0b4733db7f2c738d6de09524d492d8d"} Nov 03 04:46:50.885397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.885284 5275 scope.go:122] "RemoveContainer" containerID="de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633" Nov 03 04:46:50.895102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.895080 5275 scope.go:122] "RemoveContainer" containerID="de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633" Nov 03 04:46:50.895427 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:46:50.895404 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633\": container with ID starting with de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633 not found: ID does not exist" containerID="de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633" Nov 03 04:46:50.895545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.895436 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633"} err="failed to get container status \"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633\": rpc error: code = NotFound desc = could not find container \"de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633\": container with ID starting with de94a40345717836f26b318499ca0d7921abd62d7594061af42e8237423d7633 not found: ID does not exist" Nov 03 04:46:50.955520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.955482 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\") pod \"bd03194d-2810-4ddd-a614-4d1330517347\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " Nov 03 04:46:50.955520 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.955527 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\" (UniqueName: \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\") pod \"bd03194d-2810-4ddd-a614-4d1330517347\" (UID: \"bd03194d-2810-4ddd-a614-4d1330517347\") " Nov 03 04:46:50.955940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.955917 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume" pod "bd03194d-2810-4ddd-a614-4d1330517347" (UID: "bd03194d-2810-4ddd-a614-4d1330517347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:46:50.957832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:50.957811 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf" pod "bd03194d-2810-4ddd-a614-4d1330517347" (UID: "bd03194d-2810-4ddd-a614-4d1330517347"). InnerVolumeSpecName "kube-api-access-hhzrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:46:51.056507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:51.056457 5275 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd03194d-2810-4ddd-a614-4d1330517347-plugins-config-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:51.056507 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:51.056499 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzrf\" (UniqueName: \"kubernetes.io/projected/bd03194d-2810-4ddd-a614-4d1330517347-kube-api-access-hhzrf\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:51.300685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:51.300641 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:46:51.302790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:51.302768 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcb427-6cb4422zbg"] Nov 03 04:46:52.051176 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:52.051141 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l"] Nov 03 04:46:52.100653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:52.100611 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kpv4l"] Nov 03 04:46:52.983169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:52.983136 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a29a228-de1c-489c-be17-221960dd6600" path="/var/lib/kubelet/pods/4a29a228-de1c-489c-be17-221960dd6600/volumes" Nov 03 04:46:52.983564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:52.983517 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd03194d-2810-4ddd-a614-4d1330517347" path="/var/lib/kubelet/pods/bd03194d-2810-4ddd-a614-4d1330517347/volumes" Nov 03 04:46:55.714014 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:55.713978 5275 scope.go:122] "RemoveContainer" containerID="569402e2c73f46adb599ebcc554b8004b1ba9c8aa3d8bb01400fdca982aa4912" Nov 03 04:46:55.723454 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:46:55.723428 5275 scope.go:122] "RemoveContainer" containerID="e690ac7fdd20c04bf50536bdc9b8841a5bd30f1f6abf758c694d3f583c14113c" Nov 03 04:47:02.332586 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:47:02.332548 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:47:04.589837 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:04.589802 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:47:04.590295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:04.590162 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" containerID="cri-o://1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d" gracePeriod=30 Nov 03 04:47:06.203934 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.203868 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2"] Nov 03 04:47:06.253111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.253074 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv9t2"] Nov 03 04:47:06.327016 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.326981 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk"] Nov 03 04:47:06.327441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327425 5275 cpu_manager.go:414] "containerMap: removing container" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:47:06.327441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327441 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327465 5275 cpu_manager.go:414] "containerMap: removing container" podUID="5aa36d9f-6eca-4092-b119-1483b7e0d1ba" containerName="revoke-keys" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327473 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa36d9f-6eca-4092-b119-1483b7e0d1ba" containerName="revoke-keys" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327502 5275 cpu_manager.go:414] "containerMap: removing container" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327509 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327583 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa36d9f-6eca-4092-b119-1483b7e0d1ba" containerName="revoke-keys" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327590 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="644412fd-c2f3-4011-9824-1ff1cf84e086" containerName="cleanup" Nov 03 04:47:06.327609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.327601 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd03194d-2810-4ddd-a614-4d1330517347" containerName="payload-pre-processing" Nov 03 04:47:06.330767 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.330750 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.385643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.385602 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:47:06.385643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.385602 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:47:06.385643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.385624 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:47:06.386642 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.385665 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:47:06.389111 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.389085 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk"] Nov 03 04:47:06.407505 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.407469 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgj5\" (UniqueName: \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.407700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.407514 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.407782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.407745 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.508600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.508508 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgj5\" (UniqueName: \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.508600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.508570 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.508862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.508714 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.509295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.509269 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.511972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.511941 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.570022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.569984 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgj5\" (UniqueName: \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\") pod \"maas-api-revoke-sub-3d4156f1-4gvtk\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.643519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.643477 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:06.983048 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:06.983017 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519b6cd5-2458-429f-8220-053cca604251" path="/var/lib/kubelet/pods/519b6cd5-2458-429f-8220-053cca604251/volumes" Nov 03 04:47:07.027607 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:07.027565 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk"] Nov 03 04:47:07.029973 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:47:07.029941 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee6ba97_87a5_41c7_bdcb_a553babe25ca.slice/crio-c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388 WatchSource:0}: Error finding container c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388: Status 404 returned error can't find the container with id c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388 Nov 03 04:47:07.985644 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:07.985609 5275 generic.go:356] "Generic (PLEG): container finished" podID="eee6ba97-87a5-41c7-bdcb-a553babe25ca" containerID="4cceb8a8afa85335c4140f94db182455db5662d7615dc9e04a06d5743325ff86" exitCode=0 Nov 03 04:47:07.986195 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:07.985697 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" event={"ID":"eee6ba97-87a5-41c7-bdcb-a553babe25ca","Type":"ContainerDied","Data":"4cceb8a8afa85335c4140f94db182455db5662d7615dc9e04a06d5743325ff86"} Nov 03 04:47:07.986195 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:07.985749 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" event={"ID":"eee6ba97-87a5-41c7-bdcb-a553babe25ca","Type":"ContainerStarted","Data":"c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388"} Nov 03 04:47:09.133488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.133458 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:09.237600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.237563 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\") pod \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " Nov 03 04:47:09.237791 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.237716 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\" (UniqueName: \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\") pod \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " Nov 03 04:47:09.237840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.237808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\") pod \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\" (UID: \"eee6ba97-87a5-41c7-bdcb-a553babe25ca\") " Nov 03 04:47:09.238054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.238032 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca" pod "eee6ba97-87a5-41c7-bdcb-a553babe25ca" (UID: "eee6ba97-87a5-41c7-bdcb-a553babe25ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:47:09.238223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.238194 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.239994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.239971 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5" pod "eee6ba97-87a5-41c7-bdcb-a553babe25ca" (UID: "eee6ba97-87a5-41c7-bdcb-a553babe25ca"). InnerVolumeSpecName "kube-api-access-6kgj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:47:09.240088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.239997 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca" pod "eee6ba97-87a5-41c7-bdcb-a553babe25ca" (UID: "eee6ba97-87a5-41c7-bdcb-a553babe25ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:47:09.339205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.339165 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6ba97-87a5-41c7-bdcb-a553babe25ca-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.339205 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.339198 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kgj5\" (UniqueName: \"kubernetes.io/projected/eee6ba97-87a5-41c7-bdcb-a553babe25ca-kube-api-access-6kgj5\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.575996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.575956 5275 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:47:09.576185 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.576013 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:47:09.837258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.837231 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:47:09.946643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946610 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.946842 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946658 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.946842 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946720 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.946842 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946768 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.946842 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946808 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.947039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946916 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.947039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946934 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:09.947039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.946981 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.947039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.947253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947065 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\") pod \"660a2a7c-9766-4196-bac7-082eaaabcac8\" (UID: \"660a2a7c-9766-4196-bac7-082eaaabcac8\") " Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947339 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947328 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947551 5275 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-certs\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947572 5275 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-data\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947588 5275 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-credential-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947603 5275 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-workload-socket\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:09.947650 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.947612 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:47:09.949460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.949432 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:47:09.949460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.949447 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:47:09.949605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.949463 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:09.949605 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.949530 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l" pod "660a2a7c-9766-4196-bac7-082eaaabcac8" (UID: "660a2a7c-9766-4196-bac7-082eaaabcac8"). InnerVolumeSpecName "kube-api-access-jpb7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:47:09.998488 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.998456 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" Nov 03 04:47:09.998696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.998455 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk" event={"ID":"eee6ba97-87a5-41c7-bdcb-a553babe25ca","Type":"ContainerDied","Data":"c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388"} Nov 03 04:47:09.998696 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:09.998608 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25e729577da0623d7311bafe9ab42a96294430f04ace64afd6095401d5d7388" Nov 03 04:47:10.000460 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.000431 5275 generic.go:356] "Generic (PLEG): container finished" podID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerID="1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d" exitCode=0 Nov 03 04:47:10.000596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.000472 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerDied","Data":"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d"} Nov 03 04:47:10.000596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.000484 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" Nov 03 04:47:10.000596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.000511 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj" event={"ID":"660a2a7c-9766-4196-bac7-082eaaabcac8","Type":"ContainerDied","Data":"2e0ead7cce8116cf121edbcae62f0b89fbb390b0cf9ce8376e48af9ed15be98c"} Nov 03 04:47:10.000596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.000540 5275 scope.go:122] "RemoveContainer" containerID="1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d" Nov 03 04:47:10.011554 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.011535 5275 scope.go:122] "RemoveContainer" containerID="9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066" Nov 03 04:47:10.021013 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.020992 5275 scope.go:122] "RemoveContainer" containerID="1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d" Nov 03 04:47:10.021358 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:47:10.021331 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d\": container with ID starting with 1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d not found: ID does not exist" containerID="1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d" Nov 03 04:47:10.021448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.021367 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d"} err="failed to get container status \"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d\": rpc error: code = NotFound desc = could not find container \"1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d\": container with ID starting with 1bfbe7b6891c974e5ba06c06c633c9dbeaf425489158aee17b6ef57bdc80da0d not found: ID does not exist" Nov 03 04:47:10.021448 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.021388 5275 scope.go:122] "RemoveContainer" containerID="9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066" Nov 03 04:47:10.021764 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:47:10.021725 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066\": container with ID starting with 9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066 not found: ID does not exist" containerID="9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066" Nov 03 04:47:10.021821 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.021777 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066"} err="failed to get container status \"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066\": rpc error: code = NotFound desc = could not find container \"9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066\": container with ID starting with 9d3a3c3c61f5b6151b7d6468da21f491499af6b47da8aaf6b677e1e7e26dd066 not found: ID does not exist" Nov 03 04:47:10.048574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.048530 5275 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a2a7c-9766-4196-bac7-082eaaabcac8-istiod-ca-cert\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:10.048574 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.048573 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpb7l\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-kube-api-access-jpb7l\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:10.048782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.048591 5275 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-podinfo\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:10.048782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.048601 5275 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-token\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:10.048782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.048611 5275 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a2a7c-9766-4196-bac7-082eaaabcac8-istio-envoy\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:10.218980 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.218937 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:47:10.222960 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.222932 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcb427-openshift-default-84c6559f57q9jtj"] Nov 03 04:47:10.984534 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:10.984495 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" path="/var/lib/kubelet/pods/660a2a7c-9766-4196-bac7-082eaaabcac8/volumes" Nov 03 04:47:14.048804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:14.048770 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh"] Nov 03 04:47:14.098387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:14.098349 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhvrh"] Nov 03 04:47:14.982796 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:14.982765 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e009273-d621-440f-958e-934a41f78aa2" path="/var/lib/kubelet/pods/9e009273-d621-440f-958e-934a41f78aa2/volumes" Nov 03 04:47:25.277840 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.277796 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 04:47:25.278445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.278087 5275 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" podUID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" containerName="manager" containerID="cri-o://3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c" gracePeriod=10 Nov 03 04:47:25.539544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.539487 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 04:47:25.609618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.609581 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\" (UniqueName: \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\") pod \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " Nov 03 04:47:25.609815 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.609665 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\") pod \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\" (UID: \"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04\") " Nov 03 04:47:25.610090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.610064 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume" pod "11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" (UID: "11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:47:25.611946 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.611920 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl" pod "11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" (UID: "11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04"). InnerVolumeSpecName "kube-api-access-4tgpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:47:25.710630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.710593 5275 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-extensions-socket-volume\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:25.710630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:25.710627 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgpl\" (UniqueName: \"kubernetes.io/projected/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04-kube-api-access-4tgpl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:47:26.092694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.092656 5275 generic.go:356] "Generic (PLEG): container finished" podID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" containerID="3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c" exitCode=0 Nov 03 04:47:26.092909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.092700 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" Nov 03 04:47:26.092909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.092701 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" event={"ID":"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04","Type":"ContainerDied","Data":"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c"} Nov 03 04:47:26.092909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.092830 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn" event={"ID":"11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04","Type":"ContainerDied","Data":"ae4d49b6cd33d5ffcd7b655cb9138349e00bbc59317706ce7a04b7d2935e6976"} Nov 03 04:47:26.092909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.092891 5275 scope.go:122] "RemoveContainer" containerID="3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c" Nov 03 04:47:26.108363 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.108343 5275 scope.go:122] "RemoveContainer" containerID="3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c" Nov 03 04:47:26.108637 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:47:26.108612 5275 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c\": container with ID starting with 3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c not found: ID does not exist" containerID="3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c" Nov 03 04:47:26.108713 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.108651 5275 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c"} err="failed to get container status \"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c\": rpc error: code = NotFound desc = could not find container \"3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c\": container with ID starting with 3d7991fbfce3548824cb1de53734888ffc0339e7e61f3e6a66069cdd10a18e0c not found: ID does not exist" Nov 03 04:47:26.259071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.259037 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 04:47:26.309090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.309053 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nvhkn"] Nov 03 04:47:26.983082 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:26.983047 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" path="/var/lib/kubelet/pods/11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04/volumes" Nov 03 04:47:32.329716 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:47:32.329671 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:47:37.049730 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:37.049683 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52"] Nov 03 04:47:37.099288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:37.099244 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6lh52"] Nov 03 04:47:38.982994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:38.982959 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8ff723-519d-4c4d-b653-e8270ea15bfe" path="/var/lib/kubelet/pods/8c8ff723-519d-4c4d-b653-e8270ea15bfe/volumes" Nov 03 04:47:54.051877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:54.051825 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl"] Nov 03 04:47:54.053794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:54.053771 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-d2stl"] Nov 03 04:47:54.983385 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:54.983352 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35619080-6b6f-4cb3-9252-a8383db634ac" path="/var/lib/kubelet/pods/35619080-6b6f-4cb3-9252-a8383db634ac/volumes" Nov 03 04:47:55.755830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:55.755799 5275 scope.go:122] "RemoveContainer" containerID="56f9084f0dacd61fc25b224fc23f938adee3d84e76fd3703d02595f5cffd7c46" Nov 03 04:47:55.767822 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:55.767797 5275 scope.go:122] "RemoveContainer" containerID="111fb1abd6e70bb99f38c66f983fbf0d1eac35f941362009dace902f1e86b92b" Nov 03 04:47:55.777060 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:47:55.777040 5275 scope.go:122] "RemoveContainer" containerID="b9c6745f1cb2401b5004fa5f3b90321df2d4cffdca26773994c558fe3d92d092" Nov 03 04:48:02.327962 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:48:02.327925 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:48:28.983782 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.983682 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8"] Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984265 5275 cpu_manager.go:414] "containerMap: removing container" podUID="eee6ba97-87a5-41c7-bdcb-a553babe25ca" containerName="revoke-keys" Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984286 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee6ba97-87a5-41c7-bdcb-a553babe25ca" containerName="revoke-keys" Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984320 5275 cpu_manager.go:414] "containerMap: removing container" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984328 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984352 5275 cpu_manager.go:414] "containerMap: removing container" podUID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" containerName="manager" Nov 03 04:48:28.984437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984361 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" containerName="manager" Nov 03 04:48:28.984784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984452 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:28.984784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984463 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:28.984784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984475 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f0dd9b-eedd-4bbd-99d1-f422d2e9fc04" containerName="manager" Nov 03 04:48:28.984784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.984488 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee6ba97-87a5-41c7-bdcb-a553babe25ca" containerName="revoke-keys" Nov 03 04:48:28.987603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:28.987583 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.038401 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.038366 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8"] Nov 03 04:48:29.039918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.039891 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q9rn7\"" Nov 03 04:48:29.045528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.045494 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c034d21-2332-44f8-9625-e293270f0c89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.045671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.045636 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpd7\" (UniqueName: \"kubernetes.io/projected/5c034d21-2332-44f8-9625-e293270f0c89-kube-api-access-hjpd7\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.146442 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.146396 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c034d21-2332-44f8-9625-e293270f0c89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.146654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.146495 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpd7\" (UniqueName: \"kubernetes.io/projected/5c034d21-2332-44f8-9625-e293270f0c89-kube-api-access-hjpd7\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.146784 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.146762 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c034d21-2332-44f8-9625-e293270f0c89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.208010 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.207972 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpd7\" (UniqueName: \"kubernetes.io/projected/5c034d21-2332-44f8-9625-e293270f0c89-kube-api-access-hjpd7\") pod \"kuadrant-operator-controller-manager-678d87766f-jknk8\" (UID: \"5c034d21-2332-44f8-9625-e293270f0c89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.301531 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.301494 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:29.780029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:29.780002 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8"] Nov 03 04:48:29.782071 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:48:29.782038 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c034d21_2332_44f8_9625_e293270f0c89.slice/crio-88982a0637522ec088e7715eb99fb87e973fe008210041d824bc8548eb28e302 WatchSource:0}: Error finding container 88982a0637522ec088e7715eb99fb87e973fe008210041d824bc8548eb28e302: Status 404 returned error can't find the container with id 88982a0637522ec088e7715eb99fb87e973fe008210041d824bc8548eb28e302 Nov 03 04:48:30.462506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:30.462467 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" event={"ID":"5c034d21-2332-44f8-9625-e293270f0c89","Type":"ContainerStarted","Data":"87e1a383a015a16826fe31b8b6f5d6a2a4a1b83dde303237a7856cc80a6ed55d"} Nov 03 04:48:30.462977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:30.462545 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" event={"ID":"5c034d21-2332-44f8-9625-e293270f0c89","Type":"ContainerStarted","Data":"88982a0637522ec088e7715eb99fb87e973fe008210041d824bc8548eb28e302"} Nov 03 04:48:30.462977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:30.462605 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:30.683733 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:30.683670 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" podStartSLOduration=2.683655969 podStartE2EDuration="2.683655969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:48:30.631977506 +0000 UTC m=+6401.784665919" watchObservedRunningTime="2026-11-03 04:48:30.683655969 +0000 UTC m=+6401.836344389" Nov 03 04:48:32.329515 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:48:32.329477 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:48:41.470672 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:41.470638 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jknk8" Nov 03 04:48:53.741631 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.741595 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d"] Nov 03 04:48:53.742072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.742057 5275 cpu_manager.go:414] "containerMap: removing container" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:53.742118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.742072 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660a2a7c-9766-4196-bac7-082eaaabcac8" containerName="istio-proxy" Nov 03 04:48:53.745345 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.745329 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.801702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.801660 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:48:53.805723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.805686 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt469\" (UniqueName: \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.805918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.805746 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.805918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.805825 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.848758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.848725 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:48:53.848758 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.848740 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:48:53.849022 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.849002 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:48:53.852293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.852266 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d"] Nov 03 04:48:53.906735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.906696 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt469\" (UniqueName: \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.906963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.906742 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.906963 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.906807 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.907613 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.907590 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.909389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.909370 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:53.968256 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:53.968224 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt469\" (UniqueName: \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\") pod \"maas-api-revoke-sub-d576e68b-zl27d\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:54.058272 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:54.058236 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:54.441648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:54.441624 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d"] Nov 03 04:48:54.443414 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:48:54.443387 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213b432a_44cc_4b96_a867_4c960e873eb2.slice/crio-a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226 WatchSource:0}: Error finding container a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226: Status 404 returned error can't find the container with id a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226 Nov 03 04:48:54.627248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:54.627215 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" event={"ID":"213b432a-44cc-4b96-a867-4c960e873eb2","Type":"ContainerStarted","Data":"a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226"} Nov 03 04:48:55.635375 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:55.635282 5275 generic.go:356] "Generic (PLEG): container finished" podID="213b432a-44cc-4b96-a867-4c960e873eb2" containerID="bbd2f78e4a760b0fe5a23cce3259966632879739ba0a80f3eeb28e0d9ae10140" exitCode=0 Nov 03 04:48:55.635770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:55.635372 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" event={"ID":"213b432a-44cc-4b96-a867-4c960e873eb2","Type":"ContainerDied","Data":"bbd2f78e4a760b0fe5a23cce3259966632879739ba0a80f3eeb28e0d9ae10140"} Nov 03 04:48:55.815351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:55.815326 5275 scope.go:122] "RemoveContainer" containerID="da4a6b2326aeef56c4110477528c72a8b093fe9a642cc203d9d0aa57a5fc3556" Nov 03 04:48:55.824826 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:55.824806 5275 scope.go:122] "RemoveContainer" containerID="ce60e63c2382346f3d8e96dcf10802e3c1323aca237eab9f668531a3a87e5798" Nov 03 04:48:55.834075 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:55.834048 5275 scope.go:122] "RemoveContainer" containerID="1802bb9d6d0fe86c00c5f4fb7b951de83a3d7bcdda8f9344f4cf1f05e0cf8a67" Nov 03 04:48:56.788186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.788160 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:56.840214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.840176 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\" (UniqueName: \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\") pod \"213b432a-44cc-4b96-a867-4c960e873eb2\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " Nov 03 04:48:56.840403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.840234 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\") pod \"213b432a-44cc-4b96-a867-4c960e873eb2\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " Nov 03 04:48:56.840403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.840358 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\") pod \"213b432a-44cc-4b96-a867-4c960e873eb2\" (UID: \"213b432a-44cc-4b96-a867-4c960e873eb2\") " Nov 03 04:48:56.840722 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.840691 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca" pod "213b432a-44cc-4b96-a867-4c960e873eb2" (UID: "213b432a-44cc-4b96-a867-4c960e873eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:48:56.840875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.840811 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:48:56.842467 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.842441 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca" pod "213b432a-44cc-4b96-a867-4c960e873eb2" (UID: "213b432a-44cc-4b96-a867-4c960e873eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:48:56.842550 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.842532 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469" pod "213b432a-44cc-4b96-a867-4c960e873eb2" (UID: "213b432a-44cc-4b96-a867-4c960e873eb2"). InnerVolumeSpecName "kube-api-access-nt469". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:48:56.941732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.941642 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt469\" (UniqueName: \"kubernetes.io/projected/213b432a-44cc-4b96-a867-4c960e873eb2-kube-api-access-nt469\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:48:56.941732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:56.941675 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213b432a-44cc-4b96-a867-4c960e873eb2-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:48:57.646909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:57.646874 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" Nov 03 04:48:57.646909 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:57.646891 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d" event={"ID":"213b432a-44cc-4b96-a867-4c960e873eb2","Type":"ContainerDied","Data":"a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226"} Nov 03 04:48:57.647186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:48:57.646969 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a952df33df4db89a2bad9ca30d0a8db35f451f41bf8f44e26d1acc5402e87226" Nov 03 04:49:02.330041 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:49:02.329999 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:49:07.049042 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:07.048996 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5"] Nov 03 04:49:07.098536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:07.098496 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-z5vn5"] Nov 03 04:49:08.982913 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:08.982880 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f1c05a-8012-4f3b-bb9d-d7102932f5f8" path="/var/lib/kubelet/pods/04f1c05a-8012-4f3b-bb9d-d7102932f5f8/volumes" Nov 03 04:49:13.904883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:13.904015 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:49:13.904883 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:13.904085 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:49:14.094952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:14.092988 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:49:14.094952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:14.093101 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:49:15.697193 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.697157 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n"] Nov 03 04:49:15.697590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.697583 5275 cpu_manager.go:414] "containerMap: removing container" podUID="213b432a-44cc-4b96-a867-4c960e873eb2" containerName="revoke-keys" Nov 03 04:49:15.697634 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.697595 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213b432a-44cc-4b96-a867-4c960e873eb2" containerName="revoke-keys" Nov 03 04:49:15.697671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.697657 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="213b432a-44cc-4b96-a867-4c960e873eb2" containerName="revoke-keys" Nov 03 04:49:15.700915 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.700898 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.759339 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.759298 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n"] Nov 03 04:49:15.784965 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.784934 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.785127 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.785052 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.785188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.785172 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgs95\" (UniqueName: \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.805374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.805345 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:49:15.805374 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.805323 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:49:15.805596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.805443 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:49:15.805596 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.805444 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:49:15.886412 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.886371 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgs95\" (UniqueName: \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.886630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.886474 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.886630 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.886530 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.887249 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.887222 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.888989 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.888960 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:15.947456 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:15.947387 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgs95\" (UniqueName: \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\") pod \"maas-api-revoke-sub-d2bc93bc-8wg5n\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:16.013369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:16.013338 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:16.399004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:16.394798 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n"] Nov 03 04:49:16.408006 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:49:16.407970 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bc2c3a_b9eb_426a_aa9f_5829ccff4f78.slice/crio-ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28 WatchSource:0}: Error finding container ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28: Status 404 returned error can't find the container with id ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28 Nov 03 04:49:16.792325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:16.792285 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" event={"ID":"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78","Type":"ContainerStarted","Data":"ea2f37d7a0e3425d3455f671da6c1d2def5456516648eb079566526cc2a61ea2"} Nov 03 04:49:16.792739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:16.792348 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" event={"ID":"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78","Type":"ContainerStarted","Data":"ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28"} Nov 03 04:49:16.910426 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:16.910332 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" podStartSLOduration=1.910318156 podStartE2EDuration="1.910318156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:49:16.90759671 +0000 UTC m=+6448.060285130" watchObservedRunningTime="2026-11-03 04:49:16.910318156 +0000 UTC m=+6448.063006607" Nov 03 04:49:17.799684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:17.799646 5275 generic.go:356] "Generic (PLEG): container finished" podID="52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" containerID="ea2f37d7a0e3425d3455f671da6c1d2def5456516648eb079566526cc2a61ea2" exitCode=0 Nov 03 04:49:17.800231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:17.799692 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" event={"ID":"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78","Type":"ContainerDied","Data":"ea2f37d7a0e3425d3455f671da6c1d2def5456516648eb079566526cc2a61ea2"} Nov 03 04:49:18.054728 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:18.054651 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs"] Nov 03 04:49:18.057770 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:18.057746 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-htcfs"] Nov 03 04:49:18.949967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:18.949935 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:18.983424 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:18.983392 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbd200e-dfc5-4543-8735-45876a004d63" path="/var/lib/kubelet/pods/8bbd200e-dfc5-4543-8735-45876a004d63/volumes" Nov 03 04:49:19.122187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.122101 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\" (UniqueName: \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\") pod \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " Nov 03 04:49:19.122352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.122215 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\") pod \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " Nov 03 04:49:19.122352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.122301 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\") pod \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\" (UID: \"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78\") " Nov 03 04:49:19.122595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.122568 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca" pod "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" (UID: "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:49:19.122669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.122654 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:19.124569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.124537 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca" pod "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" (UID: "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:49:19.124738 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.124717 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95" pod "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" (UID: "52bc2c3a-b9eb-426a-aa9f-5829ccff4f78"). InnerVolumeSpecName "kube-api-access-cgs95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:49:19.223243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.223208 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgs95\" (UniqueName: \"kubernetes.io/projected/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-kube-api-access-cgs95\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:19.223243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.223239 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:19.811802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.811767 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" Nov 03 04:49:19.812029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.811770 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n" event={"ID":"52bc2c3a-b9eb-426a-aa9f-5829ccff4f78","Type":"ContainerDied","Data":"ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28"} Nov 03 04:49:19.812029 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:19.811920 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef85465580acbc64f57f3c9e54481ff15c55c1b1ce03f7bb3560cb6fb895ba28" Nov 03 04:49:32.329785 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:49:32.329740 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:49:44.095871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:44.094978 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:49:44.095871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:44.095079 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:49:54.061498 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.061421 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt"] Nov 03 04:49:54.061992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.061884 5275 cpu_manager.go:414] "containerMap: removing container" podUID="52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" containerName="revoke-keys" Nov 03 04:49:54.061992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.061897 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" containerName="revoke-keys" Nov 03 04:49:54.061992 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.061983 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" containerName="revoke-keys" Nov 03 04:49:54.065521 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.065502 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.120255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.120203 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:49:54.158624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.158590 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvqw\" (UniqueName: \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.158797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.158718 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.158797 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.158749 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.169677 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.169645 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:49:54.169677 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.169662 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:49:54.169921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.169759 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:49:54.171871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.171834 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt"] Nov 03 04:49:54.260204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.260160 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvqw\" (UniqueName: \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.260406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.260292 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.260406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.260327 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.261089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.261070 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.262962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.262939 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.321270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.321188 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvqw\" (UniqueName: \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\") pod \"maas-api-revoke-sub-ca686e02-z6gvt\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.378463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.378429 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:54.758729 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:54.758650 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt"] Nov 03 04:49:54.762298 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:49:54.762263 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91f575e_31d5_44dc_b62e_4a494ac1b65a.slice/crio-e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f WatchSource:0}: Error finding container e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f: Status 404 returned error can't find the container with id e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f Nov 03 04:49:55.132519 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:55.132479 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" event={"ID":"c91f575e-31d5-44dc-b62e-4a494ac1b65a","Type":"ContainerStarted","Data":"5a4aa7b30480ca54fa387361aa9335c93da006b5ca8044d22bdc95cb64b426ef"} Nov 03 04:49:55.132969 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:55.132535 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" event={"ID":"c91f575e-31d5-44dc-b62e-4a494ac1b65a","Type":"ContainerStarted","Data":"e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f"} Nov 03 04:49:55.249182 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:55.249074 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" podStartSLOduration=1.249053396 podStartE2EDuration="1.249053396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:49:55.248584599 +0000 UTC m=+6486.401272999" watchObservedRunningTime="2026-11-03 04:49:55.249053396 +0000 UTC m=+6486.401741819" Nov 03 04:49:55.862566 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:55.862484 5275 scope.go:122] "RemoveContainer" containerID="b714349d39fb00f6c3912472d20d3839a5cf3cdeddeb1c3dc91fab4791a86266" Nov 03 04:49:55.871491 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:55.871468 5275 scope.go:122] "RemoveContainer" containerID="4b8cb19c0a1c7b348c902703a929fac4a5aeea499ce73cd5b8ce1f07407b3d7d" Nov 03 04:49:56.139568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:56.139479 5275 generic.go:356] "Generic (PLEG): container finished" podID="c91f575e-31d5-44dc-b62e-4a494ac1b65a" containerID="5a4aa7b30480ca54fa387361aa9335c93da006b5ca8044d22bdc95cb64b426ef" exitCode=0 Nov 03 04:49:56.139568 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:56.139518 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" event={"ID":"c91f575e-31d5-44dc-b62e-4a494ac1b65a","Type":"ContainerDied","Data":"5a4aa7b30480ca54fa387361aa9335c93da006b5ca8044d22bdc95cb64b426ef"} Nov 03 04:49:57.288559 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.288532 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:57.394282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.394241 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\") pod \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " Nov 03 04:49:57.394484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.394290 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\") pod \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " Nov 03 04:49:57.394484 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.394457 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\" (UniqueName: \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\") pod \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\" (UID: \"c91f575e-31d5-44dc-b62e-4a494ac1b65a\") " Nov 03 04:49:57.394659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.394581 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca" pod "c91f575e-31d5-44dc-b62e-4a494ac1b65a" (UID: "c91f575e-31d5-44dc-b62e-4a494ac1b65a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:49:57.394891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.394869 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:57.396551 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.396525 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca" pod "c91f575e-31d5-44dc-b62e-4a494ac1b65a" (UID: "c91f575e-31d5-44dc-b62e-4a494ac1b65a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:49:57.396830 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.396810 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw" pod "c91f575e-31d5-44dc-b62e-4a494ac1b65a" (UID: "c91f575e-31d5-44dc-b62e-4a494ac1b65a"). InnerVolumeSpecName "kube-api-access-fjvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:49:57.495946 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.495910 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c91f575e-31d5-44dc-b62e-4a494ac1b65a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:57.495946 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:57.495943 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvqw\" (UniqueName: \"kubernetes.io/projected/c91f575e-31d5-44dc-b62e-4a494ac1b65a-kube-api-access-fjvqw\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:58.151538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:58.151503 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" Nov 03 04:49:58.151538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:58.151522 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt" event={"ID":"c91f575e-31d5-44dc-b62e-4a494ac1b65a","Type":"ContainerDied","Data":"e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f"} Nov 03 04:49:58.151804 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:49:58.151571 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34527feb5cc6907ae62b8faeb53846ffe12b0e3736771635c1ca454cca9b32f" Nov 03 04:50:02.329525 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:50:02.329489 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:50:04.852930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.852895 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7"] Nov 03 04:50:04.853366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.853299 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c91f575e-31d5-44dc-b62e-4a494ac1b65a" containerName="revoke-keys" Nov 03 04:50:04.853366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.853310 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91f575e-31d5-44dc-b62e-4a494ac1b65a" containerName="revoke-keys" Nov 03 04:50:04.853451 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.853408 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91f575e-31d5-44dc-b62e-4a494ac1b65a" containerName="revoke-keys" Nov 03 04:50:04.864620 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.864580 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:04.914357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.914318 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7"] Nov 03 04:50:04.961144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.961097 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:50:04.961144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.961109 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:50:04.961144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.961141 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:50:04.966253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.966230 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:50:04.977750 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.977728 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:04.977893 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.977783 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:04.977950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:04.977934 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9jk\" (UniqueName: \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.078569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.078537 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9jk\" (UniqueName: \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.078746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.078640 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.078746 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.078672 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.079314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.079294 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.081234 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.081214 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.140958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.140875 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9jk\" (UniqueName: \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\") pod \"maas-api-revoke-sub-14934818-pqhb7\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.178192 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.178162 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:05.555446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:05.555421 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7"] Nov 03 04:50:05.557126 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:50:05.557098 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff11042b_7a4b_4cd1_9ce0_185964d772ee.slice/crio-32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc WatchSource:0}: Error finding container 32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc: Status 404 returned error can't find the container with id 32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc Nov 03 04:50:06.196347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:06.196258 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" event={"ID":"ff11042b-7a4b-4cd1-9ce0-185964d772ee","Type":"ContainerStarted","Data":"d3c51e1b3e0519d6c1bec5a6277fbcfc1518797729e1d5a150ae1dc9a037592f"} Nov 03 04:50:06.196347 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:06.196322 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" event={"ID":"ff11042b-7a4b-4cd1-9ce0-185964d772ee","Type":"ContainerStarted","Data":"32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc"} Nov 03 04:50:06.314123 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:06.314068 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" podStartSLOduration=2.314050639 podStartE2EDuration="2.314050639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:50:06.311673034 +0000 UTC m=+6497.464361455" watchObservedRunningTime="2026-11-03 04:50:06.314050639 +0000 UTC m=+6497.466739089" Nov 03 04:50:07.203643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:07.203611 5275 generic.go:356] "Generic (PLEG): container finished" podID="ff11042b-7a4b-4cd1-9ce0-185964d772ee" containerID="d3c51e1b3e0519d6c1bec5a6277fbcfc1518797729e1d5a150ae1dc9a037592f" exitCode=0 Nov 03 04:50:07.204149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:07.203704 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" event={"ID":"ff11042b-7a4b-4cd1-9ce0-185964d772ee","Type":"ContainerDied","Data":"d3c51e1b3e0519d6c1bec5a6277fbcfc1518797729e1d5a150ae1dc9a037592f"} Nov 03 04:50:08.350777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.350748 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:08.510871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.510776 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\") pod \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " Nov 03 04:50:08.510871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.510823 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\") pod \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " Nov 03 04:50:08.511092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.510918 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\" (UniqueName: \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\") pod \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\" (UID: \"ff11042b-7a4b-4cd1-9ce0-185964d772ee\") " Nov 03 04:50:08.511192 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.511169 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca" pod "ff11042b-7a4b-4cd1-9ce0-185964d772ee" (UID: "ff11042b-7a4b-4cd1-9ce0-185964d772ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:50:08.513252 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.513220 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca" pod "ff11042b-7a4b-4cd1-9ce0-185964d772ee" (UID: "ff11042b-7a4b-4cd1-9ce0-185964d772ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:50:08.513349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.513295 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk" pod "ff11042b-7a4b-4cd1-9ce0-185964d772ee" (UID: "ff11042b-7a4b-4cd1-9ce0-185964d772ee"). InnerVolumeSpecName "kube-api-access-lx9jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:50:08.612039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.612002 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9jk\" (UniqueName: \"kubernetes.io/projected/ff11042b-7a4b-4cd1-9ce0-185964d772ee-kube-api-access-lx9jk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:08.612039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.612038 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:08.612039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:08.612054 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff11042b-7a4b-4cd1-9ce0-185964d772ee-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:09.216046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:09.216016 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" Nov 03 04:50:09.216231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:09.216052 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7" event={"ID":"ff11042b-7a4b-4cd1-9ce0-185964d772ee","Type":"ContainerDied","Data":"32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc"} Nov 03 04:50:09.216231 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:09.216102 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e18c4d9a393ac45bc297055e02c03ccbf34787305c130922ce6ec08fc19fcc" Nov 03 04:50:32.330432 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:50:32.330393 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:50:40.582395 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.582356 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb"] Nov 03 04:50:40.583066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.583042 5275 cpu_manager.go:414] "containerMap: removing container" podUID="ff11042b-7a4b-4cd1-9ce0-185964d772ee" containerName="revoke-keys" Nov 03 04:50:40.583161 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.583069 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff11042b-7a4b-4cd1-9ce0-185964d772ee" containerName="revoke-keys" Nov 03 04:50:40.583248 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.583234 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff11042b-7a4b-4cd1-9ce0-185964d772ee" containerName="revoke-keys" Nov 03 04:50:40.586476 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.586454 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.690653 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.690615 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:50:40.690865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.690615 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:50:40.690865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.690618 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:50:40.690865 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.690625 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:50:40.693150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.693124 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb"] Nov 03 04:50:40.699301 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.699280 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.699398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.699318 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ss7b\" (UniqueName: \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.699398 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.699347 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.800323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.800282 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.800323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.800336 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ss7b\" (UniqueName: \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.800564 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.800479 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.801251 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.801135 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.803044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.803016 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.862144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.862070 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ss7b\" (UniqueName: \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\") pod \"maas-api-revoke-sub-b63ffb7a-kmwkb\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:40.902486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:40.902452 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:41.279819 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:41.279788 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb"] Nov 03 04:50:41.282332 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:50:41.282306 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f1f075_adcd_4897_9064_86310a6b955a.slice/crio-6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e WatchSource:0}: Error finding container 6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e: Status 404 returned error can't find the container with id 6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e Nov 03 04:50:41.422056 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:41.422016 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" event={"ID":"00f1f075-adcd-4897-9064-86310a6b955a","Type":"ContainerStarted","Data":"6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e"} Nov 03 04:50:42.429204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:42.429111 5275 generic.go:356] "Generic (PLEG): container finished" podID="00f1f075-adcd-4897-9064-86310a6b955a" containerID="e8f3c8cc1d7c97ca10812261f998d447ece560cc5e882c34e4eefda9234cbb36" exitCode=0 Nov 03 04:50:42.429204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:42.429152 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" event={"ID":"00f1f075-adcd-4897-9064-86310a6b955a","Type":"ContainerDied","Data":"e8f3c8cc1d7c97ca10812261f998d447ece560cc5e882c34e4eefda9234cbb36"} Nov 03 04:50:43.570908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.570868 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:43.732186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.732101 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\") pod \"00f1f075-adcd-4897-9064-86310a6b955a\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " Nov 03 04:50:43.732186 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.732142 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\" (UniqueName: \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\") pod \"00f1f075-adcd-4897-9064-86310a6b955a\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " Nov 03 04:50:43.732393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.732288 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\") pod \"00f1f075-adcd-4897-9064-86310a6b955a\" (UID: \"00f1f075-adcd-4897-9064-86310a6b955a\") " Nov 03 04:50:43.732538 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.732505 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca" pod "00f1f075-adcd-4897-9064-86310a6b955a" (UID: "00f1f075-adcd-4897-9064-86310a6b955a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:50:43.734401 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.734377 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca" pod "00f1f075-adcd-4897-9064-86310a6b955a" (UID: "00f1f075-adcd-4897-9064-86310a6b955a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:50:43.734530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.734444 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b" pod "00f1f075-adcd-4897-9064-86310a6b955a" (UID: "00f1f075-adcd-4897-9064-86310a6b955a"). InnerVolumeSpecName "kube-api-access-5ss7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:50:43.833158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.833116 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:43.833158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.833148 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f1f075-adcd-4897-9064-86310a6b955a-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:43.833158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:43.833159 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ss7b\" (UniqueName: \"kubernetes.io/projected/00f1f075-adcd-4897-9064-86310a6b955a-kube-api-access-5ss7b\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:50:44.445977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:44.445939 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" event={"ID":"00f1f075-adcd-4897-9064-86310a6b955a","Type":"ContainerDied","Data":"6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e"} Nov 03 04:50:44.445977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:44.445961 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb" Nov 03 04:50:44.446198 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:44.445988 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d554b2ccc3b3a99d0c0d50c3027d76d5effd63b0a72503ceb6611c768e13a5e" Nov 03 04:50:55.896648 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:55.896617 5275 scope.go:122] "RemoveContainer" containerID="a28470d30eff8fcb161d96d657482776b615f0b2b9e7be84b1701a96575f4089" Nov 03 04:50:55.905859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:55.905825 5275 scope.go:122] "RemoveContainer" containerID="33fdfdc1c06ecd2f80be519a65b7e71fee66ce9fc4d08f021b3468db567e278c" Nov 03 04:50:55.914100 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:50:55.914079 5275 scope.go:122] "RemoveContainer" containerID="23f5416c55187dbc71b09894b81f8f1a221b3331f9e795f91cf3807cacb0c6be" Nov 03 04:51:02.562259 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:51:02.562214 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:51:11.048922 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:11.048858 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr"] Nov 03 04:51:11.098462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:11.098422 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5520a9f4-8xkpr"] Nov 03 04:51:12.983072 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:12.983037 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa36d9f-6eca-4092-b119-1483b7e0d1ba" path="/var/lib/kubelet/pods/5aa36d9f-6eca-4092-b119-1483b7e0d1ba/volumes" Nov 03 04:51:25.766586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.766503 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4"] Nov 03 04:51:25.767051 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.767032 5275 cpu_manager.go:414] "containerMap: removing container" podUID="00f1f075-adcd-4897-9064-86310a6b955a" containerName="revoke-keys" Nov 03 04:51:25.767106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.767053 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f1f075-adcd-4897-9064-86310a6b955a" containerName="revoke-keys" Nov 03 04:51:25.767149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.767112 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f1f075-adcd-4897-9064-86310a6b955a" containerName="revoke-keys" Nov 03 04:51:25.770652 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.770634 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.844877 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.844804 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.845074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.844972 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.845074 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.845034 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcmt\" (UniqueName: \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.874931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.874890 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:51:25.875151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.875044 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:51:25.875151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.875088 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:51:25.875151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.875117 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:51:25.877399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.877370 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4"] Nov 03 04:51:25.946158 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.946123 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.946344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.946188 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcmt\" (UniqueName: \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.946344 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.946253 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.946873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.946823 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:25.948828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:25.948806 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:26.007290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.007260 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcmt\" (UniqueName: \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\") pod \"maas-api-revoke-sub-294f0378-gvzq4\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:26.084366 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.084322 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:26.460816 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.460786 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4"] Nov 03 04:51:26.462536 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:51:26.462505 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ccd259_ca70_4322_973f_b77c2d8540b9.slice/crio-c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540 WatchSource:0}: Error finding container c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540: Status 404 returned error can't find the container with id c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540 Nov 03 04:51:26.788455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.788417 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" event={"ID":"83ccd259-ca70-4322-973f-b77c2d8540b9","Type":"ContainerStarted","Data":"e6997949aa1bcbf798e2d75030b472cca90799982c2c89d927e006675971699d"} Nov 03 04:51:26.788872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.788516 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" event={"ID":"83ccd259-ca70-4322-973f-b77c2d8540b9","Type":"ContainerStarted","Data":"c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540"} Nov 03 04:51:26.905346 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:26.905292 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" podStartSLOduration=1.905275172 podStartE2EDuration="1.905275172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:51:26.903665157 +0000 UTC m=+6578.056353578" watchObservedRunningTime="2026-11-03 04:51:26.905275172 +0000 UTC m=+6578.057963593" Nov 03 04:51:27.795351 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:27.795319 5275 generic.go:356] "Generic (PLEG): container finished" podID="83ccd259-ca70-4322-973f-b77c2d8540b9" containerID="e6997949aa1bcbf798e2d75030b472cca90799982c2c89d927e006675971699d" exitCode=0 Nov 03 04:51:27.795781 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:27.795405 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" event={"ID":"83ccd259-ca70-4322-973f-b77c2d8540b9","Type":"ContainerDied","Data":"e6997949aa1bcbf798e2d75030b472cca90799982c2c89d927e006675971699d"} Nov 03 04:51:28.937089 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:28.937060 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:29.082071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.081984 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\") pod \"83ccd259-ca70-4322-973f-b77c2d8540b9\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " Nov 03 04:51:29.082071 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.082060 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\" (UniqueName: \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\") pod \"83ccd259-ca70-4322-973f-b77c2d8540b9\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " Nov 03 04:51:29.082316 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.082199 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\") pod \"83ccd259-ca70-4322-973f-b77c2d8540b9\" (UID: \"83ccd259-ca70-4322-973f-b77c2d8540b9\") " Nov 03 04:51:29.082923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.082597 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca" pod "83ccd259-ca70-4322-973f-b77c2d8540b9" (UID: "83ccd259-ca70-4322-973f-b77c2d8540b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:51:29.082923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.082686 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:29.085084 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.085059 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca" pod "83ccd259-ca70-4322-973f-b77c2d8540b9" (UID: "83ccd259-ca70-4322-973f-b77c2d8540b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:51:29.085178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.085062 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt" pod "83ccd259-ca70-4322-973f-b77c2d8540b9" (UID: "83ccd259-ca70-4322-973f-b77c2d8540b9"). InnerVolumeSpecName "kube-api-access-wvcmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:51:29.183465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.183419 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ccd259-ca70-4322-973f-b77c2d8540b9-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:29.183465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.183457 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcmt\" (UniqueName: \"kubernetes.io/projected/83ccd259-ca70-4322-973f-b77c2d8540b9-kube-api-access-wvcmt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:29.806861 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.806824 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" event={"ID":"83ccd259-ca70-4322-973f-b77c2d8540b9","Type":"ContainerDied","Data":"c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540"} Nov 03 04:51:29.807068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.806871 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4" Nov 03 04:51:29.807068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:29.806912 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84defc3240e80f5163159e2c9e5d3900218f6f0bd4d4bee5c202ff464878540" Nov 03 04:51:32.330168 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:51:32.330130 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:51:36.521735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.521697 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d"] Nov 03 04:51:36.522228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.522176 5275 cpu_manager.go:414] "containerMap: removing container" podUID="83ccd259-ca70-4322-973f-b77c2d8540b9" containerName="revoke-keys" Nov 03 04:51:36.522228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.522189 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ccd259-ca70-4322-973f-b77c2d8540b9" containerName="revoke-keys" Nov 03 04:51:36.522307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.522265 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ccd259-ca70-4322-973f-b77c2d8540b9" containerName="revoke-keys" Nov 03 04:51:36.525208 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.525188 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.580053 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.580006 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:51:36.583214 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.583186 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d"] Nov 03 04:51:36.629414 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.629372 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:51:36.629624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.629485 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:51:36.629624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.629484 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:51:36.662747 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.662709 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.662940 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.662828 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.662994 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.662938 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksmt\" (UniqueName: \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.763813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.763772 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.764128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.763822 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.764128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.763879 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksmt\" (UniqueName: \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.764639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.764608 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.766406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.766383 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.825437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.825357 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksmt\" (UniqueName: \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\") pod \"maas-api-revoke-sub-deaf881b-w5x2d\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:36.838026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:36.837998 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:37.217786 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:37.216486 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d"] Nov 03 04:51:37.220667 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:51:37.220636 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c9d8b8_c326_4d51_82c3_42c13813b4fc.slice/crio-a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445 WatchSource:0}: Error finding container a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445: Status 404 returned error can't find the container with id a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445 Nov 03 04:51:37.859387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:37.851871 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" event={"ID":"c7c9d8b8-c326-4d51-82c3-42c13813b4fc","Type":"ContainerStarted","Data":"b10e65022c6cdf2c4d8f3f69d9fcd7c5fe95747ba4de297924528ee39e6fc966"} Nov 03 04:51:37.859387 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:37.851996 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" event={"ID":"c7c9d8b8-c326-4d51-82c3-42c13813b4fc","Type":"ContainerStarted","Data":"a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445"} Nov 03 04:51:37.969141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:37.969041 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" podStartSLOduration=1.969028726 podStartE2EDuration="1.969028726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:51:37.967824019 +0000 UTC m=+6589.120512440" watchObservedRunningTime="2026-11-03 04:51:37.969028726 +0000 UTC m=+6589.121717182" Nov 03 04:51:38.859958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:38.859918 5275 generic.go:356] "Generic (PLEG): container finished" podID="c7c9d8b8-c326-4d51-82c3-42c13813b4fc" containerID="b10e65022c6cdf2c4d8f3f69d9fcd7c5fe95747ba4de297924528ee39e6fc966" exitCode=0 Nov 03 04:51:38.860485 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:38.859975 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" event={"ID":"c7c9d8b8-c326-4d51-82c3-42c13813b4fc","Type":"ContainerDied","Data":"b10e65022c6cdf2c4d8f3f69d9fcd7c5fe95747ba4de297924528ee39e6fc966"} Nov 03 04:51:39.997031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:39.997002 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:40.101506 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.101473 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\") pod \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " Nov 03 04:51:40.101700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.101549 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\") pod \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " Nov 03 04:51:40.101700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.101659 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\" (UniqueName: \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\") pod \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\" (UID: \"c7c9d8b8-c326-4d51-82c3-42c13813b4fc\") " Nov 03 04:51:40.101942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.101915 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca" pod "c7c9d8b8-c326-4d51-82c3-42c13813b4fc" (UID: "c7c9d8b8-c326-4d51-82c3-42c13813b4fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:51:40.103908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.103871 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca" pod "c7c9d8b8-c326-4d51-82c3-42c13813b4fc" (UID: "c7c9d8b8-c326-4d51-82c3-42c13813b4fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:51:40.104059 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.104014 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt" pod "c7c9d8b8-c326-4d51-82c3-42c13813b4fc" (UID: "c7c9d8b8-c326-4d51-82c3-42c13813b4fc"). InnerVolumeSpecName "kube-api-access-pksmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:51:40.203643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.203556 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:40.203643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.203591 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:40.203643 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.203602 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksmt\" (UniqueName: \"kubernetes.io/projected/c7c9d8b8-c326-4d51-82c3-42c13813b4fc-kube-api-access-pksmt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:51:40.872364 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.872328 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" Nov 03 04:51:40.872557 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.872362 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d" event={"ID":"c7c9d8b8-c326-4d51-82c3-42c13813b4fc","Type":"ContainerDied","Data":"a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445"} Nov 03 04:51:40.872557 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:40.872442 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a983ae7e7220c75edb38a3f89432fcca7ba85bd4ae204acf8ef4af6010838445" Nov 03 04:51:50.916523 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:50.916483 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:51:50.970356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:50.970324 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:51:58.911873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:58.910984 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:51:58.911873 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:58.911076 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:51:59.087032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:59.086977 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:51:59.087411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:51:59.087366 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:52:02.330373 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:52:02.330330 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:52:05.467973 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:05.467934 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:52:05.468604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:05.468571 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:52:10.049270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:10.049230 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk"] Nov 03 04:52:10.099647 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:10.099607 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gvtk"] Nov 03 04:52:10.983321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:10.983281 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee6ba97-87a5-41c7-bdcb-a553babe25ca" path="/var/lib/kubelet/pods/eee6ba97-87a5-41c7-bdcb-a553babe25ca/volumes" Nov 03 04:52:25.091510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.091469 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj"] Nov 03 04:52:25.092033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.091921 5275 cpu_manager.go:414] "containerMap: removing container" podUID="c7c9d8b8-c326-4d51-82c3-42c13813b4fc" containerName="revoke-keys" Nov 03 04:52:25.092033 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.091934 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c9d8b8-c326-4d51-82c3-42c13813b4fc" containerName="revoke-keys" Nov 03 04:52:25.092115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.092036 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c9d8b8-c326-4d51-82c3-42c13813b4fc" containerName="revoke-keys" Nov 03 04:52:25.095604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.095585 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.149900 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.149839 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:52:25.152962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.152928 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj"] Nov 03 04:52:25.199397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.199370 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:52:25.199664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.199621 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:52:25.199730 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.199677 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:52:25.209184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.209155 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.209294 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.209274 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm46h\" (UniqueName: \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.209336 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.209311 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.310309 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.310268 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm46h\" (UniqueName: \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.310503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.310326 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.310503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.310372 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.311030 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.311009 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.312931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.312914 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.371874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.371791 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm46h\" (UniqueName: \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\") pod \"maas-api-revoke-sub-6a26528c-w2rmj\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.408704 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.408670 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:25.789948 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:25.789918 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj"] Nov 03 04:52:25.791636 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:52:25.791602 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c076ee_8d09_4ea4_aa20_a73d9210df60.slice/crio-5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478 WatchSource:0}: Error finding container 5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478: Status 404 returned error can't find the container with id 5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478 Nov 03 04:52:26.381096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:26.381004 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" event={"ID":"97c076ee-8d09-4ea4-aa20-a73d9210df60","Type":"ContainerStarted","Data":"5852839c334075645f0c0f7f1e9f424a94ba538a7251c8883e48bd3a64b2e1c6"} Nov 03 04:52:26.381096 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:26.381067 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" event={"ID":"97c076ee-8d09-4ea4-aa20-a73d9210df60","Type":"ContainerStarted","Data":"5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478"} Nov 03 04:52:26.499312 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:26.499260 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" podStartSLOduration=1.499245444 podStartE2EDuration="1.499245444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:52:26.497264484 +0000 UTC m=+6637.649952927" watchObservedRunningTime="2026-11-03 04:52:26.499245444 +0000 UTC m=+6637.651933866" Nov 03 04:52:27.388862 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:27.388813 5275 generic.go:356] "Generic (PLEG): container finished" podID="97c076ee-8d09-4ea4-aa20-a73d9210df60" containerID="5852839c334075645f0c0f7f1e9f424a94ba538a7251c8883e48bd3a64b2e1c6" exitCode=0 Nov 03 04:52:27.389288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:27.388893 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" event={"ID":"97c076ee-8d09-4ea4-aa20-a73d9210df60","Type":"ContainerDied","Data":"5852839c334075645f0c0f7f1e9f424a94ba538a7251c8883e48bd3a64b2e1c6"} Nov 03 04:52:28.543041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.543007 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:28.643942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.643903 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\" (UniqueName: \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\") pod \"97c076ee-8d09-4ea4-aa20-a73d9210df60\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " Nov 03 04:52:28.644139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.643958 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\") pod \"97c076ee-8d09-4ea4-aa20-a73d9210df60\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " Nov 03 04:52:28.644139 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.644005 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\") pod \"97c076ee-8d09-4ea4-aa20-a73d9210df60\" (UID: \"97c076ee-8d09-4ea4-aa20-a73d9210df60\") " Nov 03 04:52:28.644370 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.644340 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca" pod "97c076ee-8d09-4ea4-aa20-a73d9210df60" (UID: "97c076ee-8d09-4ea4-aa20-a73d9210df60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:52:28.646257 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.646203 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h" pod "97c076ee-8d09-4ea4-aa20-a73d9210df60" (UID: "97c076ee-8d09-4ea4-aa20-a73d9210df60"). InnerVolumeSpecName "kube-api-access-rm46h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:52:28.646369 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.646324 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca" pod "97c076ee-8d09-4ea4-aa20-a73d9210df60" (UID: "97c076ee-8d09-4ea4-aa20-a73d9210df60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:52:28.745740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.745697 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm46h\" (UniqueName: \"kubernetes.io/projected/97c076ee-8d09-4ea4-aa20-a73d9210df60-kube-api-access-rm46h\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:28.745740 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.745736 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:28.745993 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:28.745753 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c076ee-8d09-4ea4-aa20-a73d9210df60-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:29.402407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:29.402380 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" Nov 03 04:52:29.402611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:29.402421 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a26528c-w2rmj" event={"ID":"97c076ee-8d09-4ea4-aa20-a73d9210df60","Type":"ContainerDied","Data":"5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478"} Nov 03 04:52:29.402611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:29.402469 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe3e58ac42617304e61eef24f940e91ac41408f1d85e8b4f6ffae316f6d1478" Nov 03 04:52:32.329468 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:52:32.329430 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:52:35.900757 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.900715 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj"] Nov 03 04:52:35.901406 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.901381 5275 cpu_manager.go:414] "containerMap: removing container" podUID="97c076ee-8d09-4ea4-aa20-a73d9210df60" containerName="revoke-keys" Nov 03 04:52:35.901454 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.901411 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c076ee-8d09-4ea4-aa20-a73d9210df60" containerName="revoke-keys" Nov 03 04:52:35.901547 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.901536 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c076ee-8d09-4ea4-aa20-a73d9210df60" containerName="revoke-keys" Nov 03 04:52:35.905411 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.905386 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:35.961749 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:35.961710 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj"] Nov 03 04:52:36.007197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.007155 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:52:36.007197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.007193 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:52:36.007441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.007254 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:52:36.007441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.007313 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:52:36.020188 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.020150 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.020348 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.020284 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfkx\" (UniqueName: \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.020397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.020352 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.121545 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.121506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.121765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.121578 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfkx\" (UniqueName: \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.121765 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.121624 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.122422 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.122401 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.124092 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.124072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.183486 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.183410 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfkx\" (UniqueName: \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\") pod \"maas-api-revoke-sub-edb5399c-5g4fj\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.218263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.218231 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:36.596101 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:36.596077 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj"] Nov 03 04:52:36.598195 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:52:36.598167 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26696c8a_2b08_4ca6_99db_d9cd34e642d1.slice/crio-77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11 WatchSource:0}: Error finding container 77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11: Status 404 returned error can't find the container with id 77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11 Nov 03 04:52:37.449130 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:37.449097 5275 generic.go:356] "Generic (PLEG): container finished" podID="26696c8a-2b08-4ca6-99db-d9cd34e642d1" containerID="62af7d6b9783e98b93c4dfbcafaa0ddbdfb61a03a5e73e4881575117cabd8fe9" exitCode=0 Nov 03 04:52:37.449586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:37.449185 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" event={"ID":"26696c8a-2b08-4ca6-99db-d9cd34e642d1","Type":"ContainerDied","Data":"62af7d6b9783e98b93c4dfbcafaa0ddbdfb61a03a5e73e4881575117cabd8fe9"} Nov 03 04:52:37.449586 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:37.449261 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" event={"ID":"26696c8a-2b08-4ca6-99db-d9cd34e642d1","Type":"ContainerStarted","Data":"77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11"} Nov 03 04:52:38.655413 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.655046 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:38.749141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.749046 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\") pod \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " Nov 03 04:52:38.749939 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.749909 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\" (UniqueName: \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\") pod \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " Nov 03 04:52:38.750302 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.750282 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\") pod \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\" (UID: \"26696c8a-2b08-4ca6-99db-d9cd34e642d1\") " Nov 03 04:52:38.750899 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.750782 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca" pod "26696c8a-2b08-4ca6-99db-d9cd34e642d1" (UID: "26696c8a-2b08-4ca6-99db-d9cd34e642d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:52:38.751416 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.751392 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:38.752469 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.752430 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca" pod "26696c8a-2b08-4ca6-99db-d9cd34e642d1" (UID: "26696c8a-2b08-4ca6-99db-d9cd34e642d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:52:38.759368 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.759336 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx" pod "26696c8a-2b08-4ca6-99db-d9cd34e642d1" (UID: "26696c8a-2b08-4ca6-99db-d9cd34e642d1"). InnerVolumeSpecName "kube-api-access-jlfkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:52:38.853118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.853070 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfkx\" (UniqueName: \"kubernetes.io/projected/26696c8a-2b08-4ca6-99db-d9cd34e642d1-kube-api-access-jlfkx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:38.853118 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:38.853114 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26696c8a-2b08-4ca6-99db-d9cd34e642d1-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:52:39.501191 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:39.501150 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" event={"ID":"26696c8a-2b08-4ca6-99db-d9cd34e642d1","Type":"ContainerDied","Data":"77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11"} Nov 03 04:52:39.522020 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:39.521979 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77caaba054e238b93951654a05204c13c56de7fc668aad48804ade7f16c93a11" Nov 03 04:52:39.522327 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:39.522308 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb5399c-5g4fj" Nov 03 04:52:50.481270 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:50.481213 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:52:50.481947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:50.481900 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:52:55.952220 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:52:55.952139 5275 scope.go:122] "RemoveContainer" containerID="c5fecdf87f6d56f4c4d80a0f6b692986bc0a54332cd3b0349fcfa57105c11394" Nov 03 04:53:02.329892 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:53:02.329840 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:53:27.489080 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.489027 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q"] Nov 03 04:53:27.489569 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.489542 5275 cpu_manager.go:414] "containerMap: removing container" podUID="26696c8a-2b08-4ca6-99db-d9cd34e642d1" containerName="revoke-keys" Nov 03 04:53:27.489717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.489576 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26696c8a-2b08-4ca6-99db-d9cd34e642d1" containerName="revoke-keys" Nov 03 04:53:27.489785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.489716 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="26696c8a-2b08-4ca6-99db-d9cd34e642d1" containerName="revoke-keys" Nov 03 04:53:27.492896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.492874 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.551169 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.551124 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q"] Nov 03 04:53:27.597357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.597307 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:53:27.597357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.597311 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:53:27.597621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.597367 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:53:27.597621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.597319 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:53:27.599824 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.599789 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.599979 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.599871 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.600026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.600009 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.700548 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.700506 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.700773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.700601 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.700773 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.700648 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.701420 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.701397 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.703148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.703132 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.762149 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.762072 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\") pod \"maas-api-revoke-sub-75e35e8c-qvb8q\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:27.806626 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:27.806584 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:28.182337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:28.182308 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q"] Nov 03 04:53:28.184187 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:53:28.184156 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0f7c3f_5c02_467f_b135_95ed8ccd48f5.slice/crio-d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a WatchSource:0}: Error finding container d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a: Status 404 returned error can't find the container with id d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a Nov 03 04:53:28.896423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:28.896379 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" event={"ID":"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5","Type":"ContainerStarted","Data":"ba30028ec1b1ff0b88e0305a5aaa512fb76b2a0f72935faad8aa33944501dbce"} Nov 03 04:53:28.896860 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:28.896449 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" event={"ID":"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5","Type":"ContainerStarted","Data":"d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a"} Nov 03 04:53:29.014381 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:29.014328 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" podStartSLOduration=2.014312887 podStartE2EDuration="2.014312887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:53:29.012135668 +0000 UTC m=+6700.164824090" watchObservedRunningTime="2026-11-03 04:53:29.014312887 +0000 UTC m=+6700.167001308" Nov 03 04:53:29.903240 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:29.903210 5275 generic.go:356] "Generic (PLEG): container finished" podID="2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" containerID="ba30028ec1b1ff0b88e0305a5aaa512fb76b2a0f72935faad8aa33944501dbce" exitCode=0 Nov 03 04:53:29.903664 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:29.903306 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" event={"ID":"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5","Type":"ContainerDied","Data":"ba30028ec1b1ff0b88e0305a5aaa512fb76b2a0f72935faad8aa33944501dbce"} Nov 03 04:53:31.049860 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.049824 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:31.135685 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.135646 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\") pod \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " Nov 03 04:53:31.135917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.135770 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\") pod \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " Nov 03 04:53:31.135917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.135834 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\") pod \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\" (UID: \"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5\") " Nov 03 04:53:31.136238 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.136207 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca" pod "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" (UID: "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:53:31.138038 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.138007 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn" pod "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" (UID: "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5"). InnerVolumeSpecName "kube-api-access-46vfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:53:31.138128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.138084 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca" pod "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" (UID: "2c0f7c3f-5c02-467f-b135-95ed8ccd48f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:53:31.237541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.237447 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-kube-api-access-46vfn\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:31.237541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.237485 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:31.237541 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.237495 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0f7c3f-5c02-467f-b135-95ed8ccd48f5-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:31.917026 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.916996 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" Nov 03 04:53:31.917215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.916994 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e35e8c-qvb8q" event={"ID":"2c0f7c3f-5c02-467f-b135-95ed8ccd48f5","Type":"ContainerDied","Data":"d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a"} Nov 03 04:53:31.917215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:31.917115 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4eb18eb5a9d7f4023fae988f5ff74617affd6a85866ba88050067b4d517a62a" Nov 03 04:53:32.329867 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:53:32.329822 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:53:38.505090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.505050 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw"] Nov 03 04:53:38.505725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.505479 5275 cpu_manager.go:414] "containerMap: removing container" podUID="2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" containerName="revoke-keys" Nov 03 04:53:38.505725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.505491 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" containerName="revoke-keys" Nov 03 04:53:38.505725 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.505552 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0f7c3f-5c02-467f-b135-95ed8ccd48f5" containerName="revoke-keys" Nov 03 04:53:38.508717 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.508693 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.563321 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.563291 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:53:38.610968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.610933 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.611150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.611094 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.611201 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.611184 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426qk\" (UniqueName: \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.612774 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.612753 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:53:38.612872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.612772 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:53:38.612921 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.612883 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:53:38.615471 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.615452 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw"] Nov 03 04:53:38.712230 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.712195 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.712423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.712263 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426qk\" (UniqueName: \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.712423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.712307 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.712889 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.712870 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.714817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.714800 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.773788 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.773756 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-426qk\" (UniqueName: \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\") pod \"maas-api-revoke-sub-18772d3d-7qzcw\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:38.821813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:38.821780 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:39.198645 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:39.198621 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw"] Nov 03 04:53:39.201137 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:53:39.201109 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e05b09_b670_406f_94f1_0d035a7cd139.slice/crio-11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf WatchSource:0}: Error finding container 11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf: Status 404 returned error can't find the container with id 11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf Nov 03 04:53:39.965045 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:39.965009 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" event={"ID":"19e05b09-b670-406f-94f1-0d035a7cd139","Type":"ContainerStarted","Data":"e74175985a7545a618e88950ca76238ddd42d7cd572ffa2044de98daa5165811"} Nov 03 04:53:39.965461 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:39.965064 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" event={"ID":"19e05b09-b670-406f-94f1-0d035a7cd139","Type":"ContainerStarted","Data":"11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf"} Nov 03 04:53:40.082112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:40.082060 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" podStartSLOduration=2.082045774 podStartE2EDuration="2.082045774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:53:40.080295933 +0000 UTC m=+6711.232984350" watchObservedRunningTime="2026-11-03 04:53:40.082045774 +0000 UTC m=+6711.234734221" Nov 03 04:53:40.972658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:40.972615 5275 generic.go:356] "Generic (PLEG): container finished" podID="19e05b09-b670-406f-94f1-0d035a7cd139" containerID="e74175985a7545a618e88950ca76238ddd42d7cd572ffa2044de98daa5165811" exitCode=0 Nov 03 04:53:40.973125 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:40.972709 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" event={"ID":"19e05b09-b670-406f-94f1-0d035a7cd139","Type":"ContainerDied","Data":"e74175985a7545a618e88950ca76238ddd42d7cd572ffa2044de98daa5165811"} Nov 03 04:53:42.114660 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.114633 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:42.249536 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.249451 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\") pod \"19e05b09-b670-406f-94f1-0d035a7cd139\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " Nov 03 04:53:42.249706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.249553 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\" (UniqueName: \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\") pod \"19e05b09-b670-406f-94f1-0d035a7cd139\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " Nov 03 04:53:42.249706 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.249595 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\") pod \"19e05b09-b670-406f-94f1-0d035a7cd139\" (UID: \"19e05b09-b670-406f-94f1-0d035a7cd139\") " Nov 03 04:53:42.249902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.249878 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca" pod "19e05b09-b670-406f-94f1-0d035a7cd139" (UID: "19e05b09-b670-406f-94f1-0d035a7cd139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:53:42.250017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.250002 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:42.251985 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.251961 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk" pod "19e05b09-b670-406f-94f1-0d035a7cd139" (UID: "19e05b09-b670-406f-94f1-0d035a7cd139"). InnerVolumeSpecName "kube-api-access-426qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:53:42.252102 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.252041 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca" pod "19e05b09-b670-406f-94f1-0d035a7cd139" (UID: "19e05b09-b670-406f-94f1-0d035a7cd139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:53:42.351443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.351398 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e05b09-b670-406f-94f1-0d035a7cd139-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:42.351443 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.351441 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426qk\" (UniqueName: \"kubernetes.io/projected/19e05b09-b670-406f-94f1-0d035a7cd139-kube-api-access-426qk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:42.988912 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.988880 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" Nov 03 04:53:42.989228 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.988946 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18772d3d-7qzcw" event={"ID":"19e05b09-b670-406f-94f1-0d035a7cd139","Type":"ContainerDied","Data":"11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf"} Nov 03 04:53:42.989439 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:42.989418 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f581b022502061e100613b58810165b9565ce50fcce34098a1f41a550b2adf" Nov 03 04:53:49.213166 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.213123 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57"] Nov 03 04:53:49.213590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.213533 5275 cpu_manager.go:414] "containerMap: removing container" podUID="19e05b09-b670-406f-94f1-0d035a7cd139" containerName="revoke-keys" Nov 03 04:53:49.213590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.213543 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e05b09-b670-406f-94f1-0d035a7cd139" containerName="revoke-keys" Nov 03 04:53:49.213683 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.213626 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e05b09-b670-406f-94f1-0d035a7cd139" containerName="revoke-keys" Nov 03 04:53:49.216996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.216976 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.271833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.271799 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:53:49.272156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.272137 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:53:49.286474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.286440 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.286646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.286497 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdtz\" (UniqueName: \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.286825 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.286793 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.320625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.320591 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:53:49.320625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.320599 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:53:49.324609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.324583 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57"] Nov 03 04:53:49.388227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.388186 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.388497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.388285 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.388497 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.388329 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdtz\" (UniqueName: \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.388974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.388938 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.390917 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.390894 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.449954 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.449920 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdtz\" (UniqueName: \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\") pod \"maas-api-revoke-sub-500d08aa-shc57\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.530356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.530325 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:49.914529 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:49.914490 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57"] Nov 03 04:53:49.915969 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:53:49.915940 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc39f41_48a7_42cf_bf91_aa980a7dc4cd.slice/crio-99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71 WatchSource:0}: Error finding container 99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71: Status 404 returned error can't find the container with id 99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71 Nov 03 04:53:50.053530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:50.053491 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" event={"ID":"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd","Type":"ContainerStarted","Data":"99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71"} Nov 03 04:53:51.060803 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:51.060768 5275 generic.go:356] "Generic (PLEG): container finished" podID="9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" containerID="4ccdeaba36ace194e515cad0f9a0f87fdb8d58500133522d9d0d7d9d5e1604e2" exitCode=0 Nov 03 04:53:51.061258 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:51.060892 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" event={"ID":"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd","Type":"ContainerDied","Data":"4ccdeaba36ace194e515cad0f9a0f87fdb8d58500133522d9d0d7d9d5e1604e2"} Nov 03 04:53:52.207263 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.207236 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:52.316998 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.316901 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\") pod \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " Nov 03 04:53:52.317187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.317082 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\" (UniqueName: \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\") pod \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " Nov 03 04:53:52.317187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.317123 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\") pod \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\" (UID: \"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd\") " Nov 03 04:53:52.317595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.317567 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca" pod "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" (UID: "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:53:52.319289 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.319266 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz" pod "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" (UID: "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd"). InnerVolumeSpecName "kube-api-access-lwdtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:53:52.319403 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.319305 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca" pod "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" (UID: "9bc39f41-48a7-42cf-bf91-aa980a7dc4cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:53:52.418356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.418318 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:52.418356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.418353 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdtz\" (UniqueName: \"kubernetes.io/projected/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-kube-api-access-lwdtz\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:52.418356 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:52.418363 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc39f41-48a7-42cf-bf91-aa980a7dc4cd-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:53:53.073259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:53.073220 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" Nov 03 04:53:53.073259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:53.073249 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500d08aa-shc57" event={"ID":"9bc39f41-48a7-42cf-bf91-aa980a7dc4cd","Type":"ContainerDied","Data":"99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71"} Nov 03 04:53:53.073479 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:53.073295 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ab9bdb503209a46d4a2fe68b23d480d6194a7d566366dce420d2bb37425a71" Nov 03 04:53:55.977690 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:55.977653 5275 scope.go:122] "RemoveContainer" containerID="4cceb8a8afa85335c4140f94db182455db5662d7615dc9e04a06d5743325ff86" Nov 03 04:53:57.050544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:57.050512 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d"] Nov 03 04:53:57.100216 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:57.100180 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl27d"] Nov 03 04:53:58.983684 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:53:58.983648 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213b432a-44cc-4b96-a867-4c960e873eb2" path="/var/lib/kubelet/pods/213b432a-44cc-4b96-a867-4c960e873eb2/volumes" Nov 03 04:54:00.169502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.169462 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v"] Nov 03 04:54:00.169956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.169890 5275 cpu_manager.go:414] "containerMap: removing container" podUID="9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" containerName="revoke-keys" Nov 03 04:54:00.169956 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.169902 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" containerName="revoke-keys" Nov 03 04:54:00.170044 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.169977 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc39f41-48a7-42cf-bf91-aa980a7dc4cd" containerName="revoke-keys" Nov 03 04:54:00.174602 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.174580 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.231441 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.231400 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v"] Nov 03 04:54:00.277919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.277887 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:00.277919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.277840 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:00.277919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.277921 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:00.278512 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.278481 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:00.302709 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.302670 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.302938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.302717 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmlk\" (UniqueName: \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.302938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.302810 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.403609 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.403570 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.403799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.403677 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.403799 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.403703 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmlk\" (UniqueName: \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.404430 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.404404 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.406183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.406158 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.465380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.465274 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmlk\" (UniqueName: \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\") pod \"maas-api-revoke-sub-eb0fd6fc-mxb8v\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.487692 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.487657 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:00.865359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:00.865321 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v"] Nov 03 04:54:00.866654 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:00.866622 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d11e00_2ccd_4b26_955a_29b00420eb16.slice/crio-a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5 WatchSource:0}: Error finding container a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5: Status 404 returned error can't find the container with id a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5 Nov 03 04:54:01.119464 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:01.119378 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" event={"ID":"34d11e00-2ccd-4b26-955a-29b00420eb16","Type":"ContainerStarted","Data":"a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5"} Nov 03 04:54:02.126702 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:02.126667 5275 generic.go:356] "Generic (PLEG): container finished" podID="34d11e00-2ccd-4b26-955a-29b00420eb16" containerID="cbdbe8dc11ceba399eb19aa2fc68089978d14f4e75ecb7e14ced347694b5f557" exitCode=0 Nov 03 04:54:02.127133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:02.126760 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" event={"ID":"34d11e00-2ccd-4b26-955a-29b00420eb16","Type":"ContainerDied","Data":"cbdbe8dc11ceba399eb19aa2fc68089978d14f4e75ecb7e14ced347694b5f557"} Nov 03 04:54:02.330064 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:54:02.330032 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:54:03.348879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.348738 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:03.539109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.539054 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\" (UniqueName: \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\") pod \"34d11e00-2ccd-4b26-955a-29b00420eb16\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " Nov 03 04:54:03.539340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.539180 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\") pod \"34d11e00-2ccd-4b26-955a-29b00420eb16\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " Nov 03 04:54:03.539340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.539279 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\") pod \"34d11e00-2ccd-4b26-955a-29b00420eb16\" (UID: \"34d11e00-2ccd-4b26-955a-29b00420eb16\") " Nov 03 04:54:03.539864 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.539816 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca" pod "34d11e00-2ccd-4b26-955a-29b00420eb16" (UID: "34d11e00-2ccd-4b26-955a-29b00420eb16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:03.542041 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.542004 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk" pod "34d11e00-2ccd-4b26-955a-29b00420eb16" (UID: "34d11e00-2ccd-4b26-955a-29b00420eb16"). InnerVolumeSpecName "kube-api-access-zjmlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:03.542331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.542308 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca" pod "34d11e00-2ccd-4b26-955a-29b00420eb16" (UID: "34d11e00-2ccd-4b26-955a-29b00420eb16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:03.640492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.640439 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:03.640492 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.640488 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmlk\" (UniqueName: \"kubernetes.io/projected/34d11e00-2ccd-4b26-955a-29b00420eb16-kube-api-access-zjmlk\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:03.640801 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:03.640510 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d11e00-2ccd-4b26-955a-29b00420eb16-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:04.153872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:04.150219 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" event={"ID":"34d11e00-2ccd-4b26-955a-29b00420eb16","Type":"ContainerDied","Data":"a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5"} Nov 03 04:54:04.153872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:04.150302 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40df7f4d80c1ba7ad1315f430cf435f54abbc0bd5d376e719d5def70f020be5" Nov 03 04:54:04.153872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:04.150371 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0fd6fc-mxb8v" Nov 03 04:54:05.458068 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:05.458023 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:05.458723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:05.458685 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:11.075768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.075722 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd"] Nov 03 04:54:11.076308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.076172 5275 cpu_manager.go:414] "containerMap: removing container" podUID="34d11e00-2ccd-4b26-955a-29b00420eb16" containerName="revoke-keys" Nov 03 04:54:11.076308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.076184 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d11e00-2ccd-4b26-955a-29b00420eb16" containerName="revoke-keys" Nov 03 04:54:11.076308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.076267 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d11e00-2ccd-4b26-955a-29b00420eb16" containerName="revoke-keys" Nov 03 04:54:11.078968 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.078941 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.182894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.182832 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:11.182894 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.182891 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:11.183165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.182891 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:11.183165 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.182899 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:11.185438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.185418 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd"] Nov 03 04:54:11.256686 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.256651 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9gdt\" (UniqueName: \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.256887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.256704 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.256972 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.256945 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.358104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.358011 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9gdt\" (UniqueName: \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.358104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.358081 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.358324 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.358186 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.358761 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.358738 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.360828 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.360804 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.419509 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.419461 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9gdt\" (UniqueName: \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\") pod \"maas-api-revoke-sub-108e83bb-svhsd\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:11.692789 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:11.692688 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:12.084039 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:12.084008 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd"] Nov 03 04:54:12.086249 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:12.086214 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7403b7_fc99_4138_b477_35fa522df242.slice/crio-9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8 WatchSource:0}: Error finding container 9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8: Status 404 returned error can't find the container with id 9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8 Nov 03 04:54:12.256597 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:12.256562 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" event={"ID":"0a7403b7-fc99-4138-b477-35fa522df242","Type":"ContainerStarted","Data":"9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8"} Nov 03 04:54:13.264833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:13.264740 5275 generic.go:356] "Generic (PLEG): container finished" podID="0a7403b7-fc99-4138-b477-35fa522df242" containerID="2e76f4c3ca4898430d0834d51cbb068903e025ef1a2335cd4bb7f886c6d1fe9c" exitCode=0 Nov 03 04:54:13.265227 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:13.264823 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" event={"ID":"0a7403b7-fc99-4138-b477-35fa522df242","Type":"ContainerDied","Data":"2e76f4c3ca4898430d0834d51cbb068903e025ef1a2335cd4bb7f886c6d1fe9c"} Nov 03 04:54:14.793575 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.793242 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:14.845679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.845620 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\") pod \"0a7403b7-fc99-4138-b477-35fa522df242\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " Nov 03 04:54:14.847050 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.846099 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\") pod \"0a7403b7-fc99-4138-b477-35fa522df242\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " Nov 03 04:54:14.847332 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.846637 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca" pod "0a7403b7-fc99-4138-b477-35fa522df242" (UID: "0a7403b7-fc99-4138-b477-35fa522df242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:14.847534 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.847475 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\" (UniqueName: \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\") pod \"0a7403b7-fc99-4138-b477-35fa522df242\" (UID: \"0a7403b7-fc99-4138-b477-35fa522df242\") " Nov 03 04:54:14.857700 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.857645 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:14.874932 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.873009 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca" pod "0a7403b7-fc99-4138-b477-35fa522df242" (UID: "0a7403b7-fc99-4138-b477-35fa522df242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:14.879896 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.877006 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt" pod "0a7403b7-fc99-4138-b477-35fa522df242" (UID: "0a7403b7-fc99-4138-b477-35fa522df242"). InnerVolumeSpecName "kube-api-access-x9gdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:15.016923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.974061 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7403b7-fc99-4138-b477-35fa522df242-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:15.016923 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:14.974107 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9gdt\" (UniqueName: \"kubernetes.io/projected/0a7403b7-fc99-4138-b477-35fa522df242-kube-api-access-x9gdt\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:15.311658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:15.310404 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" event={"ID":"0a7403b7-fc99-4138-b477-35fa522df242","Type":"ContainerDied","Data":"9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8"} Nov 03 04:54:15.311658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:15.310473 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9643c698ce77154dfc64cc1e7dc282aa28f4a9589844133ae497bb1dd572f8" Nov 03 04:54:15.311658 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:15.310560 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108e83bb-svhsd" Nov 03 04:54:19.048920 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:19.048882 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n"] Nov 03 04:54:19.098434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:19.098401 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8wg5n"] Nov 03 04:54:20.983771 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:20.983716 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bc2c3a-b9eb-426a-aa9f-5829ccff4f78" path="/var/lib/kubelet/pods/52bc2c3a-b9eb-426a-aa9f-5829ccff4f78/volumes" Nov 03 04:54:21.882736 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.882703 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw"] Nov 03 04:54:21.883177 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.883162 5275 cpu_manager.go:414] "containerMap: removing container" podUID="0a7403b7-fc99-4138-b477-35fa522df242" containerName="revoke-keys" Nov 03 04:54:21.883229 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.883179 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7403b7-fc99-4138-b477-35fa522df242" containerName="revoke-keys" Nov 03 04:54:21.883268 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.883239 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7403b7-fc99-4138-b477-35fa522df242" containerName="revoke-keys" Nov 03 04:54:21.886694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.886676 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:21.941600 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.941540 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:21.990625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.990592 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:21.990625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.990631 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:21.991105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.990686 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:21.993277 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:21.993255 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw"] Nov 03 04:54:22.040694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.040659 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.040941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.040734 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.040941 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.040877 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599hl\" (UniqueName: \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.141712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.141613 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.141712 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.141673 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599hl\" (UniqueName: \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.142031 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.141809 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.142489 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.142466 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.144246 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.144225 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.202588 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.202558 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599hl\" (UniqueName: \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\") pod \"maas-api-revoke-sub-e564a0e2-5bqgw\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.500919 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.500781 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:22.877463 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:22.877432 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw"] Nov 03 04:54:22.879284 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:22.879255 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33adc4a_8ed2_4fed_93c9_46da4e53e48e.slice/crio-05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0 WatchSource:0}: Error finding container 05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0: Status 404 returned error can't find the container with id 05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0 Nov 03 04:54:23.372790 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:23.372750 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" event={"ID":"a33adc4a-8ed2-4fed-93c9-46da4e53e48e","Type":"ContainerStarted","Data":"b5bb15cdc424fe6b5ed1bcf7a36921d8d0f167fd726b4f59917fdd4922e8d5c1"} Nov 03 04:54:23.373319 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:23.372816 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" event={"ID":"a33adc4a-8ed2-4fed-93c9-46da4e53e48e","Type":"ContainerStarted","Data":"05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0"} Nov 03 04:54:23.489841 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:23.489737 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" podStartSLOduration=2.489724253 podStartE2EDuration="2.489724253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:54:23.488160061 +0000 UTC m=+6754.640848483" watchObservedRunningTime="2026-11-03 04:54:23.489724253 +0000 UTC m=+6754.642412675" Nov 03 04:54:24.380408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:24.380369 5275 generic.go:356] "Generic (PLEG): container finished" podID="a33adc4a-8ed2-4fed-93c9-46da4e53e48e" containerID="b5bb15cdc424fe6b5ed1bcf7a36921d8d0f167fd726b4f59917fdd4922e8d5c1" exitCode=0 Nov 03 04:54:24.380817 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:24.380458 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" event={"ID":"a33adc4a-8ed2-4fed-93c9-46da4e53e48e","Type":"ContainerDied","Data":"b5bb15cdc424fe6b5ed1bcf7a36921d8d0f167fd726b4f59917fdd4922e8d5c1"} Nov 03 04:54:25.530131 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.530106 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:25.683090 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.683001 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\") pod \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " Nov 03 04:54:25.683253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.683116 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\") pod \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " Nov 03 04:54:25.683253 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.683145 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\" (UniqueName: \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\") pod \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\" (UID: \"a33adc4a-8ed2-4fed-93c9-46da4e53e48e\") " Nov 03 04:54:25.683393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.683368 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca" pod "a33adc4a-8ed2-4fed-93c9-46da4e53e48e" (UID: "a33adc4a-8ed2-4fed-93c9-46da4e53e48e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:25.683554 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.683528 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:25.685433 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.685412 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl" pod "a33adc4a-8ed2-4fed-93c9-46da4e53e48e" (UID: "a33adc4a-8ed2-4fed-93c9-46da4e53e48e"). InnerVolumeSpecName "kube-api-access-599hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:25.685549 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.685429 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca" pod "a33adc4a-8ed2-4fed-93c9-46da4e53e48e" (UID: "a33adc4a-8ed2-4fed-93c9-46da4e53e48e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:25.785115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.785072 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:25.785115 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:25.785108 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599hl\" (UniqueName: \"kubernetes.io/projected/a33adc4a-8ed2-4fed-93c9-46da4e53e48e-kube-api-access-599hl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:26.398777 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:26.398744 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" Nov 03 04:54:26.399136 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:26.398776 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e564a0e2-5bqgw" event={"ID":"a33adc4a-8ed2-4fed-93c9-46da4e53e48e","Type":"ContainerDied","Data":"05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0"} Nov 03 04:54:26.399274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:26.399168 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a4c2acfa2d51b4ff900ef90fb77f9ea3dcbdd72a9df86d1c780a340c9bebc0" Nov 03 04:54:28.913875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:28.911989 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:28.913875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:28.912096 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:29.082552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:29.082038 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:29.082552 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:29.082108 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:32.329614 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:54:32.329578 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:54:32.590218 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.590125 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h"] Nov 03 04:54:32.590593 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.590577 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a33adc4a-8ed2-4fed-93c9-46da4e53e48e" containerName="revoke-keys" Nov 03 04:54:32.590654 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.590597 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33adc4a-8ed2-4fed-93c9-46da4e53e48e" containerName="revoke-keys" Nov 03 04:54:32.590694 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.590686 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33adc4a-8ed2-4fed-93c9-46da4e53e48e" containerName="revoke-keys" Nov 03 04:54:32.594171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.594142 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.612975 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.612945 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8zl\" (UniqueName: \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.613135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.612983 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.613135 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.613006 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.648611 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.648564 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:32.698062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.698028 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:32.698062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.698058 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:32.698062 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.698066 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:32.700688 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.700662 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h"] Nov 03 04:54:32.713544 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.713516 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8zl\" (UniqueName: \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.713641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.713552 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.713641 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.713581 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.714280 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.714262 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.716105 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.716080 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.774549 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.774514 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8zl\" (UniqueName: \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\") pod \"maas-api-revoke-sub-5a82f62b-shc4h\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:32.912546 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:32.912459 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:33.288931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:33.288888 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h"] Nov 03 04:54:33.291929 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:33.291885 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b13fb1_eebe_48f4_8e65_52602ee6de87.slice/crio-1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491 WatchSource:0}: Error finding container 1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491: Status 404 returned error can't find the container with id 1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491 Nov 03 04:54:33.479482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:33.479444 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" event={"ID":"a2b13fb1-eebe-48f4-8e65-52602ee6de87","Type":"ContainerStarted","Data":"1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491"} Nov 03 04:54:34.487608 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:34.487508 5275 generic.go:356] "Generic (PLEG): container finished" podID="a2b13fb1-eebe-48f4-8e65-52602ee6de87" containerID="a0a2aee483fb5f3001c65d319d28e02a2c60bc28059f5a631968ec6ae21fd0e5" exitCode=0 Nov 03 04:54:34.488100 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:34.487594 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" event={"ID":"a2b13fb1-eebe-48f4-8e65-52602ee6de87","Type":"ContainerDied","Data":"a0a2aee483fb5f3001c65d319d28e02a2c60bc28059f5a631968ec6ae21fd0e5"} Nov 03 04:54:35.636243 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.636219 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:35.639371 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.639352 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\") pod \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " Nov 03 04:54:35.639438 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.639393 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\" (UniqueName: \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\") pod \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " Nov 03 04:54:35.639482 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.639458 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\") pod \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\" (UID: \"a2b13fb1-eebe-48f4-8e65-52602ee6de87\") " Nov 03 04:54:35.639879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.639828 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca" pod "a2b13fb1-eebe-48f4-8e65-52602ee6de87" (UID: "a2b13fb1-eebe-48f4-8e65-52602ee6de87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:35.641687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.641665 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl" pod "a2b13fb1-eebe-48f4-8e65-52602ee6de87" (UID: "a2b13fb1-eebe-48f4-8e65-52602ee6de87"). InnerVolumeSpecName "kube-api-access-nj8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:35.641687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.641672 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca" pod "a2b13fb1-eebe-48f4-8e65-52602ee6de87" (UID: "a2b13fb1-eebe-48f4-8e65-52602ee6de87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:35.740183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.740149 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:35.740183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.740179 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b13fb1-eebe-48f4-8e65-52602ee6de87-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:35.740183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:35.740190 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8zl\" (UniqueName: \"kubernetes.io/projected/a2b13fb1-eebe-48f4-8e65-52602ee6de87-kube-api-access-nj8zl\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:36.500040 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:36.500007 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" Nov 03 04:54:36.500237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:36.500043 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a82f62b-shc4h" event={"ID":"a2b13fb1-eebe-48f4-8e65-52602ee6de87","Type":"ContainerDied","Data":"1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491"} Nov 03 04:54:36.500237 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:36.500102 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1812a262b242727b43fed71b2dec3655a815a225b5b09d57a8000088238d6491" Nov 03 04:54:43.360496 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.360450 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj"] Nov 03 04:54:43.361107 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.361086 5275 cpu_manager.go:414] "containerMap: removing container" podUID="a2b13fb1-eebe-48f4-8e65-52602ee6de87" containerName="revoke-keys" Nov 03 04:54:43.361164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.361110 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b13fb1-eebe-48f4-8e65-52602ee6de87" containerName="revoke-keys" Nov 03 04:54:43.361275 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.361263 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b13fb1-eebe-48f4-8e65-52602ee6de87" containerName="revoke-keys" Nov 03 04:54:43.364721 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.364695 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.417762 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.417725 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.417978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.417893 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pf6\" (UniqueName: \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.417978 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.417960 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.421938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.421889 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj"] Nov 03 04:54:43.466337 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.466308 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:43.466528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.466313 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:43.466528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.466315 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:43.466528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.466395 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:43.519069 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.519033 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pf6\" (UniqueName: \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.519261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.519100 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.519261 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.519150 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.519785 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.519764 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.521625 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.521596 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.580323 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.580286 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pf6\" (UniqueName: \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\") pod \"maas-api-revoke-sub-ab1f887b-fsfsj\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:43.680204 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:43.680109 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:44.056259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:44.056225 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj"] Nov 03 04:54:44.057537 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:44.057494 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74957efa_61ee_4160_8ab8_69d7d62a19c7.slice/crio-4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d WatchSource:0}: Error finding container 4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d: Status 404 returned error can't find the container with id 4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d Nov 03 04:54:44.554624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:44.554576 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" event={"ID":"74957efa-61ee-4160-8ab8-69d7d62a19c7","Type":"ContainerStarted","Data":"ffbc16ca45d9e8d462da8d248530a06cc82a4f831f7861a52514c34123f366db"} Nov 03 04:54:44.555142 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:44.554682 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" event={"ID":"74957efa-61ee-4160-8ab8-69d7d62a19c7","Type":"ContainerStarted","Data":"4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d"} Nov 03 04:54:44.671703 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:44.671598 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" podStartSLOduration=1.6715849 podStartE2EDuration="1.6715849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:54:44.669051202 +0000 UTC m=+6775.821739623" watchObservedRunningTime="2026-11-03 04:54:44.6715849 +0000 UTC m=+6775.824273311" Nov 03 04:54:45.561985 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:45.561944 5275 generic.go:356] "Generic (PLEG): container finished" podID="74957efa-61ee-4160-8ab8-69d7d62a19c7" containerID="ffbc16ca45d9e8d462da8d248530a06cc82a4f831f7861a52514c34123f366db" exitCode=0 Nov 03 04:54:45.562397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:45.561982 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" event={"ID":"74957efa-61ee-4160-8ab8-69d7d62a19c7","Type":"ContainerDied","Data":"ffbc16ca45d9e8d462da8d248530a06cc82a4f831f7861a52514c34123f366db"} Nov 03 04:54:46.778829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.778481 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:46.863109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.862996 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\" (UniqueName: \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\") pod \"74957efa-61ee-4160-8ab8-69d7d62a19c7\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " Nov 03 04:54:46.863295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.863117 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\") pod \"74957efa-61ee-4160-8ab8-69d7d62a19c7\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " Nov 03 04:54:46.863295 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.863164 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\") pod \"74957efa-61ee-4160-8ab8-69d7d62a19c7\" (UID: \"74957efa-61ee-4160-8ab8-69d7d62a19c7\") " Nov 03 04:54:46.863590 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.863557 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca" pod "74957efa-61ee-4160-8ab8-69d7d62a19c7" (UID: "74957efa-61ee-4160-8ab8-69d7d62a19c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:46.866128 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.866089 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6" pod "74957efa-61ee-4160-8ab8-69d7d62a19c7" (UID: "74957efa-61ee-4160-8ab8-69d7d62a19c7"). InnerVolumeSpecName "kube-api-access-k2pf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:46.866274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.866173 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca" pod "74957efa-61ee-4160-8ab8-69d7d62a19c7" (UID: "74957efa-61ee-4160-8ab8-69d7d62a19c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:46.979902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.976500 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:46.979902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.976545 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74957efa-61ee-4160-8ab8-69d7d62a19c7-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:46.979902 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:46.976562 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pf6\" (UniqueName: \"kubernetes.io/projected/74957efa-61ee-4160-8ab8-69d7d62a19c7-kube-api-access-k2pf6\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:47.610871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:47.607146 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" event={"ID":"74957efa-61ee-4160-8ab8-69d7d62a19c7","Type":"ContainerDied","Data":"4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d"} Nov 03 04:54:47.610871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:47.607214 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc2b2300d8b34de944306e220fd66d86c3f5442ae2437f03edea3eefe2b9b2d" Nov 03 04:54:47.610871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:47.607286 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1f887b-fsfsj" Nov 03 04:54:48.458871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:48.451942 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vbhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:48.458871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:48.452012 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vbhnb" podUID="73b572c1-6ea2-4a48-9633-41d7b8966db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:54.127244 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.127201 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g"] Nov 03 04:54:54.127833 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.127813 5275 cpu_manager.go:414] "containerMap: removing container" podUID="74957efa-61ee-4160-8ab8-69d7d62a19c7" containerName="revoke-keys" Nov 03 04:54:54.127976 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.127835 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74957efa-61ee-4160-8ab8-69d7d62a19c7" containerName="revoke-keys" Nov 03 04:54:54.128047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.128024 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="74957efa-61ee-4160-8ab8-69d7d62a19c7" containerName="revoke-keys" Nov 03 04:54:54.131530 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.131507 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.189676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.189644 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g"] Nov 03 04:54:54.197202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.197168 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.197404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.197280 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzlb\" (UniqueName: \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.197404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.197327 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.234875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.234812 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:54:54.234875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.234832 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:54:54.234875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.234875 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:54:54.235133 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.234836 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:54:54.297918 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.297883 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzlb\" (UniqueName: \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.298112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.297957 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.298112 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.298025 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.298737 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.298716 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.300417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.300395 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.358982 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.358953 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzlb\" (UniqueName: \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\") pod \"maas-api-revoke-sub-c27a8804-wmj2g\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.444774 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.444693 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:54.825197 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:54.825167 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g"] Nov 03 04:54:54.826791 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:54:54.826759 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f60164_bc42_4cb2_be7f_112221a2991c.slice/crio-bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e WatchSource:0}: Error finding container bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e: Status 404 returned error can't find the container with id bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e Nov 03 04:54:55.669242 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:55.669200 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" event={"ID":"40f60164-bc42-4cb2-be7f-112221a2991c","Type":"ContainerStarted","Data":"a41f24d9a79e1261a01a55d8c443cfa941c8b51a224e479551d550323d701bac"} Nov 03 04:54:55.669669 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:55.669271 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" event={"ID":"40f60164-bc42-4cb2-be7f-112221a2991c","Type":"ContainerStarted","Data":"bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e"} Nov 03 04:54:55.785349 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:55.784827 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" podStartSLOduration=1.784810182 podStartE2EDuration="1.784810182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:54:55.784357697 +0000 UTC m=+6786.937046117" watchObservedRunningTime="2026-11-03 04:54:55.784810182 +0000 UTC m=+6786.937498605" Nov 03 04:54:56.001146 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:56.001055 5275 scope.go:122] "RemoveContainer" containerID="bbd2f78e4a760b0fe5a23cce3259966632879739ba0a80f3eeb28e0d9ae10140" Nov 03 04:54:56.678735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:56.677916 5275 generic.go:356] "Generic (PLEG): container finished" podID="40f60164-bc42-4cb2-be7f-112221a2991c" containerID="a41f24d9a79e1261a01a55d8c443cfa941c8b51a224e479551d550323d701bac" exitCode=0 Nov 03 04:54:56.678735 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:56.678092 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" event={"ID":"40f60164-bc42-4cb2-be7f-112221a2991c","Type":"ContainerDied","Data":"a41f24d9a79e1261a01a55d8c443cfa941c8b51a224e479551d550323d701bac"} Nov 03 04:54:58.230001 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.229907 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt"] Nov 03 04:54:58.295875 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.283322 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca686e02-z6gvt"] Nov 03 04:54:58.465871 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.465466 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:58.637938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.616260 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\") pod \"40f60164-bc42-4cb2-be7f-112221a2991c\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " Nov 03 04:54:58.637938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.616421 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\") pod \"40f60164-bc42-4cb2-be7f-112221a2991c\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " Nov 03 04:54:58.637938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.616475 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\" (UniqueName: \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\") pod \"40f60164-bc42-4cb2-be7f-112221a2991c\" (UID: \"40f60164-bc42-4cb2-be7f-112221a2991c\") " Nov 03 04:54:58.637938 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.625074 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca" pod "40f60164-bc42-4cb2-be7f-112221a2991c" (UID: "40f60164-bc42-4cb2-be7f-112221a2991c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:54:58.726156 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.718218 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:58.770065 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.769994 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca" pod "40f60164-bc42-4cb2-be7f-112221a2991c" (UID: "40f60164-bc42-4cb2-be7f-112221a2991c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:54:58.788879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.772343 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" event={"ID":"40f60164-bc42-4cb2-be7f-112221a2991c","Type":"ContainerDied","Data":"bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e"} Nov 03 04:54:58.788879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.772402 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc289fec94f0ee76e7d84172acff92e8028adac9697fff6478d6c4c73cd1484e" Nov 03 04:54:58.788879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.772471 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c27a8804-wmj2g" Nov 03 04:54:58.788879 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.787417 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb" pod "40f60164-bc42-4cb2-be7f-112221a2991c" (UID: "40f60164-bc42-4cb2-be7f-112221a2991c"). InnerVolumeSpecName "kube-api-access-dpzlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:54:58.836057 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.836016 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f60164-bc42-4cb2-be7f-112221a2991c-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:58.837285 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.837259 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzlb\" (UniqueName: \"kubernetes.io/projected/40f60164-bc42-4cb2-be7f-112221a2991c-kube-api-access-dpzlb\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:54:58.926930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.922968 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nxgbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:58.926930 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:58.923044 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nxgbf" podUID="21c9c1c8-26c9-4b51-abb9-a86c517ac991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:59.016874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:59.016209 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91f575e-31d5-44dc-b62e-4a494ac1b65a" path="/var/lib/kubelet/pods/c91f575e-31d5-44dc-b62e-4a494ac1b65a/volumes" Nov 03 04:54:59.171465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:59.170969 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gc8n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:59.171465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:54:59.171037 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gc8n2" podUID="c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:55:02.330400 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:55:02.330363 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:55:04.902834 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.902797 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm"] Nov 03 04:55:04.903446 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.903423 5275 cpu_manager.go:414] "containerMap: removing container" podUID="40f60164-bc42-4cb2-be7f-112221a2991c" containerName="revoke-keys" Nov 03 04:55:04.903540 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.903448 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f60164-bc42-4cb2-be7f-112221a2991c" containerName="revoke-keys" Nov 03 04:55:04.903594 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.903569 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f60164-bc42-4cb2-be7f-112221a2991c" containerName="revoke-keys" Nov 03 04:55:04.907054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.907031 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:04.962178 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:04.962118 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:55:05.001453 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.001422 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.001618 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.001483 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.001687 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.001648 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtwx\" (UniqueName: \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.011999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.011966 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:55:05.011999 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.011968 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:55:05.012164 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.012046 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:55:05.014903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.014877 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm"] Nov 03 04:55:05.102621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.102590 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.102621 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.102632 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.102874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.102670 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtwx\" (UniqueName: \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.103352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.103323 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.105160 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.105142 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.164657 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.164578 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtwx\" (UniqueName: \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\") pod \"maas-api-revoke-sub-92577f7b-mvqjm\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.220397 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.220356 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:05.596502 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.596452 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm"] Nov 03 04:55:05.598265 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:55:05.598236 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb837fe_fd56_48bb_b497_502ff6e40c0c.slice/crio-bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61 WatchSource:0}: Error finding container bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61: Status 404 returned error can't find the container with id bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61 Nov 03 04:55:05.819308 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:05.819269 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" event={"ID":"8eb837fe-fd56-48bb-b497-502ff6e40c0c","Type":"ContainerStarted","Data":"bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61"} Nov 03 04:55:06.826903 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:06.826866 5275 generic.go:356] "Generic (PLEG): container finished" podID="8eb837fe-fd56-48bb-b497-502ff6e40c0c" containerID="686b2596820a47517ca1076a53d1400baf810d961be0e5d5945d3704a4889fb5" exitCode=0 Nov 03 04:55:06.827333 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:06.826937 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" event={"ID":"8eb837fe-fd56-48bb-b497-502ff6e40c0c","Type":"ContainerDied","Data":"686b2596820a47517ca1076a53d1400baf810d961be0e5d5945d3704a4889fb5"} Nov 03 04:55:08.042298 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.042261 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:08.133058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.132946 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\") pod \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " Nov 03 04:55:08.133058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.133022 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\") pod \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " Nov 03 04:55:08.133325 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.133164 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\" (UniqueName: \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\") pod \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\" (UID: \"8eb837fe-fd56-48bb-b497-502ff6e40c0c\") " Nov 03 04:55:08.133444 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.133411 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca" pod "8eb837fe-fd56-48bb-b497-502ff6e40c0c" (UID: "8eb837fe-fd56-48bb-b497-502ff6e40c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:55:08.133576 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.133555 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:08.139359 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.139323 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca" pod "8eb837fe-fd56-48bb-b497-502ff6e40c0c" (UID: "8eb837fe-fd56-48bb-b497-502ff6e40c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:55:08.139595 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.139364 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx" pod "8eb837fe-fd56-48bb-b497-502ff6e40c0c" (UID: "8eb837fe-fd56-48bb-b497-502ff6e40c0c"). InnerVolumeSpecName "kube-api-access-vxtwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:55:08.234379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.234326 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb837fe-fd56-48bb-b497-502ff6e40c0c-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:08.234379 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.234375 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtwx\" (UniqueName: \"kubernetes.io/projected/8eb837fe-fd56-48bb-b497-502ff6e40c0c-kube-api-access-vxtwx\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:08.855151 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.855110 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" event={"ID":"8eb837fe-fd56-48bb-b497-502ff6e40c0c","Type":"ContainerDied","Data":"bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61"} Nov 03 04:55:08.855445 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.855425 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb38ee75c5f51436226a263119458ca73af000ae741d86a316ff5642be0a7f61" Nov 03 04:55:08.855668 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:08.855651 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92577f7b-mvqjm" Nov 03 04:55:09.051960 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:09.051923 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7"] Nov 03 04:55:09.069436 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:09.069392 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14934818-pqhb7"] Nov 03 04:55:10.983380 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:10.983346 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff11042b-7a4b-4cd1-9ce0-185964d772ee" path="/var/lib/kubelet/pods/ff11042b-7a4b-4cd1-9ce0-185964d772ee/volumes" Nov 03 04:55:15.663305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.663268 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887"] Nov 03 04:55:15.663812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.663677 5275 cpu_manager.go:414] "containerMap: removing container" podUID="8eb837fe-fd56-48bb-b497-502ff6e40c0c" containerName="revoke-keys" Nov 03 04:55:15.663812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.663688 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb837fe-fd56-48bb-b497-502ff6e40c0c" containerName="revoke-keys" Nov 03 04:55:15.663812 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.663779 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb837fe-fd56-48bb-b497-502ff6e40c0c" containerName="revoke-keys" Nov 03 04:55:15.667183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.667165 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.687305 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.687268 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.687481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.687404 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.687481 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.687458 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkww\" (UniqueName: \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.721341 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.721304 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:55:15.770961 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.770923 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:55:15.771184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.770972 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:55:15.771184 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.771059 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:55:15.774967 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.774938 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887"] Nov 03 04:55:15.789047 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.789013 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.789217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.789097 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.789217 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.789136 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkww\" (UniqueName: \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.789705 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.789682 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.791671 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.791650 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.850343 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.850303 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkww\" (UniqueName: \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\") pod \"maas-api-revoke-sub-7f66d3e9-ps887\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:15.980752 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:15.980655 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:16.359987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:16.359955 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887"] Nov 03 04:55:16.361963 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:55:16.361922 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d406bb_7d22_4c80_9dc4_437791c03dfd.slice/crio-104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806 WatchSource:0}: Error finding container 104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806: Status 404 returned error can't find the container with id 104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806 Nov 03 04:55:16.922340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:16.922244 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" event={"ID":"b6d406bb-7d22-4c80-9dc4-437791c03dfd","Type":"ContainerStarted","Data":"2b1d735a8763357a3706f94db331dbab07e81640271cbff89bbedbbe21a1d236"} Nov 03 04:55:16.922340 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:16.922311 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" event={"ID":"b6d406bb-7d22-4c80-9dc4-437791c03dfd","Type":"ContainerStarted","Data":"104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806"} Nov 03 04:55:17.040950 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:17.040891 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" podStartSLOduration=2.040876696 podStartE2EDuration="2.040876696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:55:17.03815199 +0000 UTC m=+6808.190840413" watchObservedRunningTime="2026-11-03 04:55:17.040876696 +0000 UTC m=+6808.193565114" Nov 03 04:55:17.930175 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:17.930136 5275 generic.go:356] "Generic (PLEG): container finished" podID="b6d406bb-7d22-4c80-9dc4-437791c03dfd" containerID="2b1d735a8763357a3706f94db331dbab07e81640271cbff89bbedbbe21a1d236" exitCode=0 Nov 03 04:55:17.930659 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:17.930224 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" event={"ID":"b6d406bb-7d22-4c80-9dc4-437791c03dfd","Type":"ContainerDied","Data":"2b1d735a8763357a3706f94db331dbab07e81640271cbff89bbedbbe21a1d236"} Nov 03 04:55:19.147947 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.147907 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:19.255931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.255239 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\") pod \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " Nov 03 04:55:19.255931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.255300 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\") pod \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " Nov 03 04:55:19.255931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.255363 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\" (UniqueName: \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\") pod \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\" (UID: \"b6d406bb-7d22-4c80-9dc4-437791c03dfd\") " Nov 03 04:55:19.256887 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.256829 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca" pod "b6d406bb-7d22-4c80-9dc4-437791c03dfd" (UID: "b6d406bb-7d22-4c80-9dc4-437791c03dfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:55:19.279971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.270934 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca" pod "b6d406bb-7d22-4c80-9dc4-437791c03dfd" (UID: "b6d406bb-7d22-4c80-9dc4-437791c03dfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:55:19.279971 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.273974 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww" pod "b6d406bb-7d22-4c80-9dc4-437791c03dfd" (UID: "b6d406bb-7d22-4c80-9dc4-437791c03dfd"). InnerVolumeSpecName "kube-api-access-2lkww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:55:19.372996 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.372947 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:19.373279 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.373256 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d406bb-7d22-4c80-9dc4-437791c03dfd-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:19.373434 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:19.373419 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lkww\" (UniqueName: \"kubernetes.io/projected/b6d406bb-7d22-4c80-9dc4-437791c03dfd-kube-api-access-2lkww\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:20.019874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:20.014730 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" event={"ID":"b6d406bb-7d22-4c80-9dc4-437791c03dfd","Type":"ContainerDied","Data":"104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806"} Nov 03 04:55:20.019874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:20.014818 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="104c9d41d0b2fbcb6d216fda59c49f57814c322c4197fdcd62366ad97d7cc806" Nov 03 04:55:20.019874 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:20.014954 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f66d3e9-ps887" Nov 03 04:55:20.511870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:20.511302 5275 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fl4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:55:20.511870 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:20.511413 5275 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fl4d7" podUID="519d3784-84ae-4be3-9404-536cc811d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:55:26.428389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.428344 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl"] Nov 03 04:55:26.429004 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.428985 5275 cpu_manager.go:414] "containerMap: removing container" podUID="b6d406bb-7d22-4c80-9dc4-437791c03dfd" containerName="revoke-keys" Nov 03 04:55:26.429088 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.429008 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d406bb-7d22-4c80-9dc4-437791c03dfd" containerName="revoke-keys" Nov 03 04:55:26.429172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.429161 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d406bb-7d22-4c80-9dc4-437791c03dfd" containerName="revoke-keys" Nov 03 04:55:26.433144 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.433120 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.488639 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.488601 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl"] Nov 03 04:55:26.533802 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.533766 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:55:26.534017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.533805 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 04:55:26.534017 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.533839 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 04:55:26.534563 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.534550 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:55:26.629745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.629696 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrx9\" (UniqueName: \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.629745 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.629743 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.630054 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.629898 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.731393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.731291 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.731393 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.731391 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrx9\" (UniqueName: \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.731662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.731409 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.732134 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.732112 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.733882 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.733860 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:26.792624 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:26.792582 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrx9\" (UniqueName: \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\") pod \"maas-api-revoke-sub-348a8d57-mtnvl\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:27.047171 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:27.047137 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:27.423908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:27.423879 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl"] Nov 03 04:55:27.425807 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:55:27.425775 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c2b323_5f48_4277_a3d7_30d3f7cb0535.slice/crio-599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179 WatchSource:0}: Error finding container 599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179: Status 404 returned error can't find the container with id 599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179 Nov 03 04:55:28.089113 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:28.089074 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" event={"ID":"29c2b323-5f48-4277-a3d7-30d3f7cb0535","Type":"ContainerStarted","Data":"5dfdc05549490d00efb11b6e202ece2095d1e9f276d14ddea5e262baf04a8181"} Nov 03 04:55:28.089604 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:28.089141 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" event={"ID":"29c2b323-5f48-4277-a3d7-30d3f7cb0535","Type":"ContainerStarted","Data":"599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179"} Nov 03 04:55:28.207353 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:28.207284 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" podStartSLOduration=2.207269749 podStartE2EDuration="2.207269749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:55:28.20419733 +0000 UTC m=+6819.356885753" watchObservedRunningTime="2026-11-03 04:55:28.207269749 +0000 UTC m=+6819.359958170" Nov 03 04:55:29.121572 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:29.121535 5275 generic.go:356] "Generic (PLEG): container finished" podID="29c2b323-5f48-4277-a3d7-30d3f7cb0535" containerID="5dfdc05549490d00efb11b6e202ece2095d1e9f276d14ddea5e262baf04a8181" exitCode=0 Nov 03 04:55:29.122271 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:29.122240 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" event={"ID":"29c2b323-5f48-4277-a3d7-30d3f7cb0535","Type":"ContainerDied","Data":"5dfdc05549490d00efb11b6e202ece2095d1e9f276d14ddea5e262baf04a8181"} Nov 03 04:55:30.936603 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.936266 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:30.975891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.975021 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\" (UniqueName: \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\") pod \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " Nov 03 04:55:30.975891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.975133 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\") pod \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " Nov 03 04:55:30.975891 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.975284 5275 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\") pod \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\" (UID: \"29c2b323-5f48-4277-a3d7-30d3f7cb0535\") " Nov 03 04:55:30.995331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.982217 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca" pod "29c2b323-5f48-4277-a3d7-30d3f7cb0535" (UID: "29c2b323-5f48-4277-a3d7-30d3f7cb0535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:55:30.995331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:30.993033 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca" pod "29c2b323-5f48-4277-a3d7-30d3f7cb0535" (UID: "29c2b323-5f48-4277-a3d7-30d3f7cb0535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:55:31.021052 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.020974 5275 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9" pod "29c2b323-5f48-4277-a3d7-30d3f7cb0535" (UID: "29c2b323-5f48-4277-a3d7-30d3f7cb0535"). InnerVolumeSpecName "kube-api-access-xjrx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:55:31.077990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.077956 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-xks-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:31.077990 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.077989 5275 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrx9\" (UniqueName: \"kubernetes.io/projected/29c2b323-5f48-4277-a3d7-30d3f7cb0535-kube-api-access-xjrx9\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:31.078215 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.078003 5275 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c2b323-5f48-4277-a3d7-30d3f7cb0535-maas-api-openshift-ca\") on node \"ip-10-0-136-7.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:55:31.151150 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.151111 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" event={"ID":"29c2b323-5f48-4277-a3d7-30d3f7cb0535","Type":"ContainerDied","Data":"599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179"} Nov 03 04:55:31.151328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.151170 5275 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="599ec2270b360e460d751fc77b19cd544fb6acdb50e7600a201cfaf752ba2179" Nov 03 04:55:31.151328 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:31.151169 5275 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348a8d57-mtnvl" Nov 03 04:55:32.329300 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:55:32.329267 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:55:44.050832 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:44.050798 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb"] Nov 03 04:55:44.055104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:44.055075 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63ffb7a-kmwkb"] Nov 03 04:55:44.982741 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:44.982697 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f1f075-adcd-4897-9064-86310a6b955a" path="/var/lib/kubelet/pods/00f1f075-adcd-4897-9064-86310a6b955a/volumes" Nov 03 04:55:56.024058 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:56.023978 5275 scope.go:122] "RemoveContainer" containerID="ea2f37d7a0e3425d3455f671da6c1d2def5456516648eb079566526cc2a61ea2" Nov 03 04:55:56.033829 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:56.033806 5275 scope.go:122] "RemoveContainer" containerID="5a4aa7b30480ca54fa387361aa9335c93da006b5ca8044d22bdc95cb64b426ef" Nov 03 04:55:59.482783 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:55:59.482747 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c66d77cd-926q9_178bdf09-2334-43af-93f3-175fc348c04a/authorino/0.log" Nov 03 04:56:02.331679 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:56:02.331645 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:56:12.709522 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:12.709489 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l6m8z_c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec/manager/0.log" Nov 03 04:56:14.420942 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:14.420909 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lskgg_f0d5b72d-3781-4776-8dfe-4b2728d5d42e/server/0.log" Nov 03 04:56:16.834974 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:16.834945 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-108e83bb-svhsd_0a7403b7-fc99-4138-b477-35fa522df242/revoke-keys/0.log" Nov 03 04:56:17.067148 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:17.067119 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-18772d3d-7qzcw_19e05b09-b670-406f-94f1-0d035a7cd139/revoke-keys/0.log" Nov 03 04:56:17.294106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:17.294079 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-294f0378-gvzq4_83ccd259-ca70-4322-973f-b77c2d8540b9/revoke-keys/0.log" Nov 03 04:56:17.523167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:17.523133 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-348a8d57-mtnvl_29c2b323-5f48-4277-a3d7-30d3f7cb0535/revoke-keys/0.log" Nov 03 04:56:17.757985 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:17.757906 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-500d08aa-shc57_9bc39f41-48a7-42cf-bf91-aa980a7dc4cd/revoke-keys/0.log" Nov 03 04:56:17.987977 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:17.987950 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5a82f62b-shc4h_a2b13fb1-eebe-48f4-8e65-52602ee6de87/revoke-keys/0.log" Nov 03 04:56:18.201494 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:18.201465 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6a26528c-w2rmj_97c076ee-8d09-4ea4-aa20-a73d9210df60/revoke-keys/0.log" Nov 03 04:56:18.441266 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:18.441238 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-75e35e8c-qvb8q_2c0f7c3f-5c02-467f-b135-95ed8ccd48f5/revoke-keys/0.log" Nov 03 04:56:18.677357 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:18.677324 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7f66d3e9-ps887_b6d406bb-7d22-4c80-9dc4-437791c03dfd/revoke-keys/0.log" Nov 03 04:56:18.909143 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:18.909111 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-92577f7b-mvqjm_8eb837fe-fd56-48bb-b497-502ff6e40c0c/revoke-keys/0.log" Nov 03 04:56:19.159183 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:19.159156 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab1f887b-fsfsj_74957efa-61ee-4160-8ab8-69d7d62a19c7/revoke-keys/0.log" Nov 03 04:56:19.382297 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:19.382266 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c27a8804-wmj2g_40f60164-bc42-4cb2-be7f-112221a2991c/revoke-keys/0.log" Nov 03 04:56:19.611807 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:19.611769 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-deaf881b-w5x2d_c7c9d8b8-c326-4d51-82c3-42c13813b4fc/revoke-keys/0.log" Nov 03 04:56:19.825663 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:19.825636 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e564a0e2-5bqgw_a33adc4a-8ed2-4fed-93c9-46da4e53e48e/revoke-keys/0.log" Nov 03 04:56:20.068293 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:20.068268 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eb0fd6fc-mxb8v_34d11e00-2ccd-4b26-955a-29b00420eb16/revoke-keys/0.log" Nov 03 04:56:20.285167 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:20.285141 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-edb5399c-5g4fj_26696c8a-2b08-4ca6-99db-d9cd34e642d1/revoke-keys/0.log" Nov 03 04:56:22.592400 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:22.592366 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c66d77cd-926q9_178bdf09-2334-43af-93f3-175fc348c04a/authorino/0.log" Nov 03 04:56:23.910714 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:23.910684 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jknk8_5c034d21-2332-44f8-9625-e293270f0c89/manager/0.log" Nov 03 04:56:24.540501 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:24.540469 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z9kkw_a60f9acc-e4f3-46e1-9951-f0fb96818e0d/manager/0.log" Nov 03 04:56:25.587290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:25.587253 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-wnpll_50cb5187-a18d-4f4b-87b9-e5a5c3c81873/istio-proxy/0.log" Nov 03 04:56:27.514759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:27.514722 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fl4d7_519d3784-84ae-4be3-9404-536cc811d02a/istio-proxy/0.log" Nov 03 04:56:27.796856 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:27.796823 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-gc8n2_c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb/istio-proxy/0.log" Nov 03 04:56:28.655962 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:28.655924 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-nxgbf_21c9c1c8-26c9-4b51-abb9-a86c517ac991/istio-proxy/0.log" Nov 03 04:56:29.049290 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:29.049252 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4"] Nov 03 04:56:29.098462 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:29.098427 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294f0378-gvzq4"] Nov 03 04:56:29.167723 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:29.167691 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-vbhnb_73b572c1-6ea2-4a48-9633-41d7b8966db8/istio-proxy/0.log" Nov 03 04:56:29.413890 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:29.413781 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn_67d09bf2-ec59-49a4-8470-a956eefdded7/istio-proxy/0.log" Nov 03 04:56:30.984404 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:30.984363 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ccd259-ca70-4322-973f-b77c2d8540b9" path="/var/lib/kubelet/pods/83ccd259-ca70-4322-973f-b77c2d8540b9/volumes" Nov 03 04:56:31.256223 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:31.256137 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8pvbf_65265c30-274d-45b6-9222-05907af3e477/main/0.log" Nov 03 04:56:32.099636 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:32.099604 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s2skg_8531b165-d56f-4bdf-893a-b2395e9dd85e/main/0.log" Nov 03 04:56:32.329640 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:56:32.329599 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:56:32.378389 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:32.378313 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b888sjr_a3cdd903-fe5b-4b17-b2b9-ee9e0535cd5e/main/0.log" Nov 03 04:56:32.933759 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:32.933731 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-m6xn2_ec639888-2e6f-40e9-899d-7409f7aef369/main/0.log" Nov 03 04:56:40.096355 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:40.096313 5275 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d"] Nov 03 04:56:40.099069 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:40.099037 5275 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deaf881b-w5x2d"] Nov 03 04:56:40.983528 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:40.983481 5275 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c9d8b8-c326-4d51-82c3-42c13813b4fc" path="/var/lib/kubelet/pods/c7c9d8b8-c326-4d51-82c3-42c13813b4fc/volumes" Nov 03 04:56:44.109372 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.109330 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmhht/must-gather-msgct"] Nov 03 04:56:44.109794 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.109776 5275 cpu_manager.go:414] "containerMap: removing container" podUID="29c2b323-5f48-4277-a3d7-30d3f7cb0535" containerName="revoke-keys" Nov 03 04:56:44.109872 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.109794 5275 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c2b323-5f48-4277-a3d7-30d3f7cb0535" containerName="revoke-keys" Nov 03 04:56:44.109958 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.109945 5275 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c2b323-5f48-4277-a3d7-30d3f7cb0535" containerName="revoke-keys" Nov 03 04:56:44.113408 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.113391 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.157987 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.157958 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-must-gather-output\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.158172 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.158007 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n99\" (UniqueName: \"kubernetes.io/projected/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-kube-api-access-n7n99\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.170046 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.170014 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmhht\"/\"kube-root-ca.crt\"" Nov 03 04:56:44.219225 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.219194 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmhht\"/\"openshift-service-ca.crt\"" Nov 03 04:56:44.219461 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.219428 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmhht\"/\"default-dockercfg-fcb9b\"" Nov 03 04:56:44.222274 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.222244 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmhht/must-gather-msgct"] Nov 03 04:56:44.259189 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.259146 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-must-gather-output\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.259390 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.259221 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n99\" (UniqueName: \"kubernetes.io/projected/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-kube-api-access-n7n99\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.259510 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.259491 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-must-gather-output\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.320122 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.320089 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n99\" (UniqueName: \"kubernetes.io/projected/ac1e70c7-a4c0-4d8f-9b63-128418b43a06-kube-api-access-n7n99\") pod \"must-gather-msgct\" (UID: \"ac1e70c7-a4c0-4d8f-9b63-128418b43a06\") " pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.427490 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.427403 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmhht/must-gather-msgct" Nov 03 04:56:44.804255 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.804222 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmhht/must-gather-msgct"] Nov 03 04:56:44.805902 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:56:44.805841 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac1e70c7_a4c0_4d8f_9b63_128418b43a06.slice/crio-712d029c88f7cc2a82018218e644d2a63a720407eeb87d034f60e1d19cb0a31a WatchSource:0}: Error finding container 712d029c88f7cc2a82018218e644d2a63a720407eeb87d034f60e1d19cb0a31a: Status 404 returned error can't find the container with id 712d029c88f7cc2a82018218e644d2a63a720407eeb87d034f60e1d19cb0a31a Nov 03 04:56:44.807859 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:44.807825 5275 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:56:45.662906 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:45.662837 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmhht/must-gather-msgct" event={"ID":"ac1e70c7-a4c0-4d8f-9b63-128418b43a06","Type":"ContainerStarted","Data":"712d029c88f7cc2a82018218e644d2a63a720407eeb87d034f60e1d19cb0a31a"} Nov 03 04:56:50.969945 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:50.969916 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:56:51.026952 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:51.026928 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvtt8_a75ca389-0c41-4d4f-9788-da6f8bb2cc70/ovn-acl-logging/0.log" Nov 03 04:56:51.712503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:51.712458 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmhht/must-gather-msgct" event={"ID":"ac1e70c7-a4c0-4d8f-9b63-128418b43a06","Type":"ContainerStarted","Data":"b090a37123495733eab02b8cfc331cced2b5ef33a02913a64c0ab24f55bc297e"} Nov 03 04:56:51.712503 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:51.712511 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmhht/must-gather-msgct" event={"ID":"ac1e70c7-a4c0-4d8f-9b63-128418b43a06","Type":"ContainerStarted","Data":"d1570d21fe39269b4cf3bacccd5f5c35bb2adc37b459c48e93589665b2b44861"} Nov 03 04:56:51.829679 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:51.829605 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kmhht/must-gather-msgct" podStartSLOduration=1.5770097280000002 podStartE2EDuration="7.829586782s" totalImagesPullingTime="6.252577054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:56:51.828174701 +0000 UTC m=+6902.980863124" watchObservedRunningTime="2026-11-03 04:56:51.829586782 +0000 UTC m=+6902.982275203" Nov 03 04:56:52.793437 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:52.793392 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4vzrm_ba38cb3f-dfe8-4af7-b185-943337c4759e/global-pull-secret-syncer/0.log" Nov 03 04:56:55.773754 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:55.773714 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l6m8z_c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec/manager/0.log" Nov 03 04:56:56.066317 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:56.066225 5275 scope.go:122] "RemoveContainer" containerID="e8f3c8cc1d7c97ca10812261f998d447ece560cc5e882c34e4eefda9234cbb36" Nov 03 04:56:56.079661 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:56.079637 5275 scope.go:122] "RemoveContainer" containerID="d3c51e1b3e0519d6c1bec5a6277fbcfc1518797729e1d5a150ae1dc9a037592f" Nov 03 04:56:56.858535 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:56.858502 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lskgg_f0d5b72d-3781-4776-8dfe-4b2728d5d42e/server/0.log" Nov 03 04:56:56.874399 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:56.874367 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v499x_4b95e773-2411-4fbc-b0f1-0e5fbde12efb/konnectivity-agent/0.log" Nov 03 04:56:57.226417 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:56:57.226334 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-7.us-west-1.compute.internal_b7e862e4bcfcee2fc388ea5939653511/haproxy/0.log" Nov 03 04:57:01.546710 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:01.546676 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l6m8z_c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec/manager/0.log" Nov 03 04:57:01.715307 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:01.715268 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c66d77cd-926q9_178bdf09-2334-43af-93f3-175fc348c04a/authorino/0.log" Nov 03 04:57:02.332214 ip-10-0-136-7 kubenswrapper[5275]: E1103 04:57:02.332176 5275 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mtw4b" containerName="extractor" Nov 03 04:57:02.898224 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.898187 5275 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq"] Nov 03 04:57:02.904085 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.904059 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:02.956259 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.956227 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"kube-root-ca.crt\"" Nov 03 04:57:02.957676 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.957652 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-proc\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:02.957739 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.957687 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-lib-modules\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:02.957783 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.957765 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-sys\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:02.957819 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.957786 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdp9\" (UniqueName: \"kubernetes.io/projected/d000b6ea-f285-4ab7-b482-660c9b141d03-kube-api-access-lxdp9\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:02.957901 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:02.957887 5275 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-podres\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.004880 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.004830 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lskgg_f0d5b72d-3781-4776-8dfe-4b2728d5d42e/server/0.log" Nov 03 04:57:03.005808 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.005780 5275 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ljm7\"/\"default-dockercfg-w89j4\"" Nov 03 04:57:03.005981 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.005833 5275 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"openshift-service-ca.crt\"" Nov 03 04:57:03.010514 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.010487 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq"] Nov 03 04:57:03.059187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059146 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-sys\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059187 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059191 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdp9\" (UniqueName: \"kubernetes.io/projected/d000b6ea-f285-4ab7-b482-660c9b141d03-kube-api-access-lxdp9\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059225 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-podres\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059281 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-sys\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059296 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-proc\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059343 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-proc\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059356 5275 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-lib-modules\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059425 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059400 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-podres\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.059662 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.059513 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d000b6ea-f285-4ab7-b482-660c9b141d03-lib-modules\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.120465 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.120426 5275 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdp9\" (UniqueName: \"kubernetes.io/projected/d000b6ea-f285-4ab7-b482-660c9b141d03-kube-api-access-lxdp9\") pod \"perf-node-gather-daemonset-grdlq\" (UID: \"d000b6ea-f285-4ab7-b482-660c9b141d03\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.218827 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.218738 5275 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.605423 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.605387 5275 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq"] Nov 03 04:57:03.606722 ip-10-0-136-7 kubenswrapper[5275]: W1103 04:57:03.606688 5275 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd000b6ea_f285_4ab7_b482_660c9b141d03.slice/crio-f5032c02043b037e5e5165f86596f44617d97dbcfc4285bce39bcf5ad9274c4d WatchSource:0}: Error finding container f5032c02043b037e5e5165f86596f44617d97dbcfc4285bce39bcf5ad9274c4d: Status 404 returned error can't find the container with id f5032c02043b037e5e5165f86596f44617d97dbcfc4285bce39bcf5ad9274c4d Nov 03 04:57:03.788109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.788070 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" event={"ID":"d000b6ea-f285-4ab7-b482-660c9b141d03","Type":"ContainerStarted","Data":"33e6f53ea3667efa5f3a7a4156a67d9798855583a6e35c50e5f2f41b455c3548"} Nov 03 04:57:03.788331 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.788133 5275 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" event={"ID":"d000b6ea-f285-4ab7-b482-660c9b141d03","Type":"ContainerStarted","Data":"f5032c02043b037e5e5165f86596f44617d97dbcfc4285bce39bcf5ad9274c4d"} Nov 03 04:57:03.788440 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.788325 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:03.905377 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:03.905280 5275 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" podStartSLOduration=1.90526398 podStartE2EDuration="1.90526398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 04:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:57:03.903470877 +0000 UTC m=+6915.056159301" watchObservedRunningTime="2026-11-03 04:57:03.90526398 +0000 UTC m=+6915.057952406" Nov 03 04:57:04.344124 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:04.344086 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jknk8_5c034d21-2332-44f8-9625-e293270f0c89/manager/0.log" Nov 03 04:57:05.246352 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:05.246317 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z9kkw_a60f9acc-e4f3-46e1-9951-f0fb96818e0d/manager/0.log" Nov 03 04:57:07.788282 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:07.788249 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l6m8z_c1eafc7c-7fa8-4b09-8cb5-0852f9b7d4ec/manager/0.log" Nov 03 04:57:08.846073 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:08.846041 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lskgg_f0d5b72d-3781-4776-8dfe-4b2728d5d42e/server/0.log" Nov 03 04:57:09.808768 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:09.808738 5275 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-grdlq" Nov 03 04:57:11.087288 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:11.087253 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bdb75_07201023-0dbc-4723-a3d1-d055fa9be1dc/kube-state-metrics/0.log" Nov 03 04:57:11.493007 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:11.492928 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bdb75_07201023-0dbc-4723-a3d1-d055fa9be1dc/kube-rbac-proxy-main/0.log" Nov 03 04:57:11.905732 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:11.905697 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bdb75_07201023-0dbc-4723-a3d1-d055fa9be1dc/kube-rbac-proxy-self/0.log" Nov 03 04:57:12.740681 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:12.740648 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2wc4b_81eb8632-4e51-4b33-bb06-fb6b3fbfd6ad/monitoring-plugin/0.log" Nov 03 04:57:13.173610 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:13.173566 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgf2f_f0f384fb-bf15-4268-a97f-e0d537873319/node-exporter/0.log" Nov 03 04:57:13.585474 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:13.585424 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgf2f_f0f384fb-bf15-4268-a97f-e0d537873319/kube-rbac-proxy/0.log" Nov 03 04:57:13.609407 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:13.609381 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-wnpll_50cb5187-a18d-4f4b-87b9-e5a5c3c81873/istio-proxy/0.log" Nov 03 04:57:14.001455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:14.001382 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgf2f_f0f384fb-bf15-4268-a97f-e0d537873319/init-textfile/0.log" Nov 03 04:57:16.713455 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:16.713427 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fl4d7_519d3784-84ae-4be3-9404-536cc811d02a/istio-proxy/0.log" Nov 03 04:57:16.929329 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:16.929303 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-gc8n2_c1ee6ca5-6b92-40cd-8d22-e90006a2e0bb/istio-proxy/0.log" Nov 03 04:57:17.573202 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:17.573147 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-nxgbf_21c9c1c8-26c9-4b51-abb9-a86c517ac991/istio-proxy/0.log" Nov 03 04:57:18.002032 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:18.001938 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-vbhnb_73b572c1-6ea2-4a48-9633-41d7b8966db8/istio-proxy/0.log" Nov 03 04:57:18.186106 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:18.186055 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/prometheus/0.log" Nov 03 04:57:18.212180 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:18.212148 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-2trvn_67d09bf2-ec59-49a4-8470-a956eefdded7/istio-proxy/0.log" Nov 03 04:57:18.598360 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:18.598326 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/config-reloader/0.log" Nov 03 04:57:19.013109 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:19.013016 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/thanos-sidecar/0.log" Nov 03 04:57:19.426104 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:19.426074 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/kube-rbac-proxy-web/0.log" Nov 03 04:57:19.843066 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:19.843041 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/kube-rbac-proxy/0.log" Nov 03 04:57:20.259141 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:20.259055 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/kube-rbac-proxy-thanos/0.log" Nov 03 04:57:20.673646 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:20.673616 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25711346-0358-49b3-839c-fd65d3686ba2/init-config-reloader/0.log" Nov 03 04:57:23.601543 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:23.601466 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/thanos-query/0.log" Nov 03 04:57:24.054813 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:24.054780 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c66d77cd-926q9_178bdf09-2334-43af-93f3-175fc348c04a/authorino/0.log" Nov 03 04:57:24.093190 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:24.093157 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/kube-rbac-proxy-web/0.log" Nov 03 04:57:24.521391 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:24.521363 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/kube-rbac-proxy/0.log" Nov 03 04:57:24.934076 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:24.933999 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/prom-label-proxy/0.log" Nov 03 04:57:24.971314 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:24.971257 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jknk8_5c034d21-2332-44f8-9625-e293270f0c89/manager/0.log" Nov 03 04:57:25.371931 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:25.371896 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/kube-rbac-proxy-rules/0.log" Nov 03 04:57:25.444769 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:25.444729 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z9kkw_a60f9acc-e4f3-46e1-9951-f0fb96818e0d/manager/0.log" Nov 03 04:57:25.760995 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:25.760908 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-wnpll_50cb5187-a18d-4f4b-87b9-e5a5c3c81873/istio-proxy/0.log" Nov 03 04:57:25.785908 ip-10-0-136-7 kubenswrapper[5275]: I1103 04:57:25.785880 5275 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cb88f77ff-knhmq_92cf23f3-5db2-49c6-9576-a824fa35da0c/kube-rbac-proxy-metrics/0.log"